The best way to control Claude Code, Codex and more from any computer or phone

main
mdx 179 lines 7.35 KB
Raw
1 ---
2 title: Looping the Generation of IaC and SDKs
3 date: 2026-07-02T16:00:00Z
4 excerpt: AI agents write resources and live tests, run them against the real API, and every unmatched error becomes a typed patch to our generated SDK. One run of this loop grew Alchemy's Cloudflare provider from 22 resources to 230 and patched 720 SDK operations, most of it in 22 hours.
5 ---
6
7 import LoopDiagram from "../../../components/LoopDiagram.astro";
8
9 <LoopDiagram />
10
11 [Alchemy](/) is a new Infrastructure-as-Code
12 framework built from the ground up in TypeScript and Effect.
13 Everything is typed code, from a resource's inputs and outputs
14 down to the SDK calls and errors inside its lifecycle logic.
15
16 Built from the ground up also means starting from zero
17 coverage. Alchemy can't reuse the providers of existing IaC
18 ecosystems like Terraform or Pulumi, and we're a team of
19 three, so hand-writing lifecycle code for thousands of cloud
20 resources isn't an option. We generate it with AI, using the
21 loop in the diagram above.
22
23 Fleets of AI agents write the resources and their live tests,
24 then run them against a real Cloudflare account. Every API
25 behavior the SDK's types don't capture becomes a patch to the
26 SDK, which regenerates and the tests rerun. One run of this
27 loop grew Alchemy's Cloudflare provider from **22 resources to
28 230** between June 11 and June 17, producing 288 test files
29 with roughly 700 tests and **720 SDK patches**. Most of it
30 landed in a single PR
31 ([#601](https://github.com/alchemy-run/alchemy/pull/601))
32 that added 101,178 lines across 542 files and was open for 22
33 hours. The PR description, in its entirety:
34
35 > Using Fable to generate all missing cloudflare resources
36
37 Deterministic codegen has been
38 producing SDKs and IaC providers from API specs for years, and
39 AI makes it even cheaper. The problem is that code generated
40 from a spec is only as truthful as the spec, and real APIs
41 drift from their specs. They return undocumented error codes,
42 misuse HTTP statuses, and null out fields the schema says are
43 required. That's why the interesting number isn't the 230
44 resources. It's the 720 patches, each one a place where a live
45 test caught the API doing something its spec doesn't admit to.
46
47 ## A software development flywheel
48
49 Alchemy's resources are built on
50 [distilled](https://github.com/alchemy-run/distilled), our
51 Effect-native SDK generated from Cloudflare's official
52 TypeScript SDK. It covers 114 services and 2,092 operations,
53 and each operation can carry a patch, a small JSON file that
54 corrects the spec where reality disagrees with it.
55
56 The run operated on one rule: **agents never handle an unknown
57 error in their own code.** An unmatched error is never caught
58 in the resource. It becomes an SDK patch and a regeneration.
59 Untyped failure handling is where AI-generated code usually
60 goes wrong, matching message substrings or guessing status
61 codes. The rule takes that option away because the code
62 doesn't typecheck until the SDK knows about the error. It also
63 makes the work compound. Every red test either fixes a
64 resource or improves the SDK for the next one.
65
66 ## One turn of the flywheel
67
68 Infrastructure-as-code is mostly disciplined error handling.
69 Deletes tolerate already-gone. Creates absorb losing a race.
70 Reads ride out eventual consistency. In Alchemy that
71 discipline is typed:
72
73 ```typescript
74 // idempotent delete: reading back an already-gone widget is success
75 turnstile.getWidget({ accountId, sitekey }).pipe(
76 Effect.catchTag("WidgetNotFound", () => Effect.succeed(undefined)),
77 );
78
79 // R2's endpoint lags a fresh bucket create, so retry the not-found until it settles
80 r2.listBucketDomainCustoms({ accountId, bucketName }).pipe(
81 Effect.retry({
82 while: (e) => e._tag === "NoSuchBucket",
83 schedule: Schedule.exponential("100 millis").pipe(
84 Schedule.both(Schedule.recurs(5)),
85 ),
86 }),
87 );
88 ```
89
90 The agents write this code first, then run it against the real
91 API. A Turnstile test deletes a widget and reads it back to
92 prove it's gone. The read fails with code `10404`, which the
93 spec doesn't declare. The `catchTag` doesn't typecheck because
94 `WidgetNotFound` isn't in the error union yet, and the only
95 permitted fix is a patch in distilled:
96
97 ```json
98 // submodules/distilled/packages/cloudflare/patches/turnstile/getWidget.json
99 {
100 "errors": {
101 "WidgetNotFound": [{ "code": 10404 }, { "code": 10407 }],
102 "Forbidden": [{ "status": 403 }]
103 }
104 }
105 ```
106
107 Regenerate the service (`bun scripts/generate.ts --service
108 turnstile`) and the union names what the API actually does:
109
110 ```typescript
111 export type GetWidgetError = DefaultErrors | WidgetNotFound | Forbidden;
112 ```
113
114 The `catchTag` compiles and the test goes green. The loop
115 continues through reconcile, read, diff, delete, and list
116 until the whole resource passes against the live API. The
117 resource is the product and the truthful SDK is the byproduct.
118 The patch lives in the SDK, not Alchemy, so anyone who uses
119 distilled's Turnstile module from now on gets `WidgetNotFound`
120 as a typed, catchable error.
121
122 ## What 1,087 patches look like
123
124 By beta.56 the corpus stood at 1,087 patched operations,
125 covering 52% of the SDK across 94 of 114 services:
126
127 | | count |
128 | --- | --- |
129 | patches declaring typed errors | 1,070 |
130 | patches fixing a response schema | 173 |
131 | patches fixing a request schema | 26 |
132 | distinct error tags introduced | 425 |
133 | matchers on Cloudflare error `code` | 1,367 |
134 | matchers on HTTP `status` | 994 |
135 | matchers needing `message.includes` | 377 |
136 | matchers needing a regex | 0 |
137
138 Most tags are the vocabulary lifecycle code needs, like
139 `Forbidden`, `WorkerNotFound`, and `NoSuchBucket`. The rest
140 catalog how APIs drift from their docs. Snippets returns 400
141 instead of 404 for a missing snippet, so the matcher reads the
142 message. Queues consumers support `http_pull` but the spec's
143 enum doesn't include it. Account settings come back null where
144 the schema says they can't. None of this is discoverable by
145 reading documentation.
146
147 ## The factory
148
149 Around twelve agents ran concurrently, each owning one
150 distilled service. Only the owner of `turnstile` writes to
151 `patches/turnstile/`, so agents never race the generator.
152 Every test follows one shape: deploy, verify out-of-band by
153 querying the API directly through distilled, mutate, verify
154 again, destroy, and prove the destroy by watching the resource
155 disappear. A test never trusts the deploy's own report of
156 success. It confirms every step against what the API actually
157 returns.
158
159 Two choices kept the loop fast. distilled is a git submodule
160 inside alchemy's bun workspace, so an agent patches the SDK
161 and the resource that consumes it in one working tree, and a
162 wave lands as two PRs, one to alchemy and one to
163 distilled. And vitest resolves distilled from its TypeScript
164 source, so a regenerated service is visible to the very next
165 test run. The loop's cycle time is the test's runtime.
166
167 ## What's next
168
169 We're scaling this loop to every cloud provider, with two
170 goals: 100% IaC coverage in Alchemy, and refined SDKs whose
171 types align with each API's real behavior rather than just its
172 spec. AWS is next, then Railway, GCP, and Azure. The
173 Cloudflare run was the proof, and it shipped in
174 [2.0.0-beta.56](/blog/2026-06-17-beta-56).
175
176 - [2.0.0-beta.56 — the release these resources shipped in](/blog/2026-06-17-beta-56)
177 - [PR #601 — the generation wave](https://github.com/alchemy-run/alchemy/pull/601)
178 - [distilled — the generated, patchable SDK](https://github.com/alchemy-run/distilled)
179 - [Cloudflare provider reference](/providers/cloudflare)