@devries-cameron20 / t3code Mirror · read-only
Fork 0

The best way to control Claude Code, Codex and more from any computer or phone

Mirrored from https://github.com/Camo67/t3code.git

Synced Nov 21, 06:00 UTC

main
mdx 273 lines 10.1 KB
Raw
1 ---
2 title: Effect AI
3 description: Wire Effect's LanguageModel and Chat services into a Cloudflare Worker — read API keys with effect/Config, provide the model layer to your handler, plug in persistence.
4 sidebar:
5 order: 5
6 ---
7
8 Effect's [`@effect/ai`](https://effect.website/docs/ai/introduction)
9 package gives you a single `LanguageModel.LanguageModel` service that
10 every model provider (OpenAI, Anthropic, Workers AI, …) ships as a
11 `Layer`. Once that layer is in scope, the rest of the surface
12 (`generateText`, `streamText`, `Toolkit`, `Chat`, …) is
13 provider-agnostic.
14
15 This guide is about the *Alchemy* side of that picture: how to build
16 the `LanguageModel` layer inside a Worker or Lambda
17 ([Runtime](/infrastructure-as-effects/runtime))
18 (`Cloudflare.Worker`, `AWS.Lambda.Function`), where to read the API
19 key from, and how to plug `Chat.Persistence` into a backing store
20 that matches the platform. For the AI calls themselves —
21 `LanguageModel.generateText`, `streamText`, `Toolkit`, structured
22 outputs — refer to
23 [Effect's AI documentation](https://effect.website/docs/ai/introduction).
24
25 ## The pattern
26
27 Every Effect AI provider is two stacked layers:
28
29 1. A **client** Layer that holds the API key and HTTP transport
30 (`OpenAiClient.layer({ apiKey })`, `AnthropicClient.layer({ apiKey })`, …).
31 2. A **language model** Layer that picks the model and any defaults
32 (`OpenAiLanguageModel.layer({ model: "gpt-4o-mini" })`, …).
33
34 You build that stack once in the Runtime's **Construction phase** and
35 `Effect.provide(...)` it on the handler:
36
37 ```typescript
38 Effect.gen(function* () {
39 // 1. Construct: build the LanguageModel layer (deploy bindings happen here).
40 const languageModel = ...;
41
42 return {
43 // 2. Exec: per-request handler with the model available.
44 fetch: Effect.gen(function* () {
45 const response = yield* LanguageModel.generateText({ prompt });
46 return HttpServerResponse.json({ text: response.text });
47 }).pipe(Effect.provide(languageModel)),
48 };
49 });
50 ```
51
52 `fetch` is just one example — the same `Effect.provide(languageModel)`
53 pattern works in a Lambda handler, an HTTP API endpoint, an RPC
54 procedure, a Workflow, or any other Effect.
55
56 ## Read the API key with `Config.Redacted`
57
58 Every upstream provider's client Layer (`OpenAiClient.layer`,
59 `AnthropicClient.layer`, …) takes an `apiKey: Redacted<string>`.
60 [`Config.Redacted`](/environments/secrets) resolved in the Runtime's
61 Construction phase gives you exactly that `Redacted<string>` — and Alchemy
62 automatically binds the value as a `secret_text` binding
63 (Cloudflare) or environment variable (Lambda) at deploy time:
64
65 ```typescript
66 import * as Config from "effect/Config";
67
68 // constructor — resolves the value AND records the binding
69 const apiKey = yield* Config.Redacted("OPENAI_API_KEY");
70 ```
71
72 `Config.Redacted("OPENAI_API_KEY")` reads `OPENAI_API_KEY` from your
73 `ConfigProvider` (e.g. `.env`) at deploy time, records the binding,
74 and resolves from that binding again at runtime — one line, both
75 phases. See [Concepts › Secrets and Config](/environments/secrets) for
76 the Construction/Runtime split, `.env`, and transformations.
77
78 Because the value is in scope right in the constructor, the model Layer can be
79 built eagerly — no `Layer.unwrap` needed:
80
81 ```typescript
82 import * as Layer from "effect/Layer";
83
84 const languageModel = OpenAiLanguageModel.layer({
85 model: "gpt-4o-mini",
86 }).pipe(
87 Layer.provide(OpenAiClient.layer({ apiKey })),
88 Layer.provide(FetchHttpClient.layer),
89 );
90 ```
91
92 ## In a Cloudflare Worker
93
94 `FetchHttpClient.layer` plugs the runtime `fetch` API into the
95 Effect HTTP client that every provider's client Layer depends on:
96
97 ```typescript
98 // src/Worker.ts
99 import * as Cloudflare from "@alchemy.run/cloudflare";
100 import * as Config from "effect/Config";
101 import * as Effect from "effect/Effect";
102 import * as Layer from "effect/Layer";
103 import { LanguageModel } from "effect/ai";
104 import * as FetchHttpClient from "effect/http/FetchHttpClient";
105 import * as HttpServerResponse from "effect/http/HttpServerResponse";
106 import { OpenAiClient, OpenAiLanguageModel } from "@effect/ai-openai";
107
108 export default Cloudflare.Worker(
109 "Worker",
110 { main: import.meta.url },
111 Effect.gen(function* () {
112 const apiKey = yield* Config.Redacted("OPENAI_API_KEY");
113
114 const languageModel = OpenAiLanguageModel.layer({
115 model: "gpt-4o-mini",
116 }).pipe(
117 Layer.provide(OpenAiClient.layer({ apiKey })),
118 Layer.provide(FetchHttpClient.layer),
119 );
120
121 return {
122 fetch: Effect.gen(function* () {
123 const response = yield* LanguageModel.generateText({
124 prompt: "Say hello.",
125 }).pipe(Effect.orDie);
126 return yield* HttpServerResponse.json({ text: response.text });
127 }).pipe(Effect.provide(languageModel)),
128 };
129 }),
130 );
131 ```
132
133 ## Persisting chats
134
135 For multi-turn conversations, Effect's `Chat` service holds the
136 turn history and exposes the same `generateText` / `streamText`
137 surface. `Chat.layerPersisted` wraps it with a
138 `Chat.Persistence` interface that needs a `BackingPersistence`
139 Layer underneath:
140
141 ```typescript
142 import { Chat } from "effect/ai";
143 import * as Persistence from "effect/persistence/Persistence";
144
145 // inside your handler:
146 const chat = yield* (yield* Chat.Persistence).getOrCreate("session-id");
147 yield* chat.generateText({ prompt: "Hello again." });
148 ```
149
150 The `BackingPersistence` Layer is what makes the chat history
151 durable. Pick the one that matches your platform:
152
153 | Platform | Backing |
154 | --- | --- |
155 | Cloudflare Worker + Durable Object | `Cloudflare.AI.DurableObjectChatPersistence` — bytes live in `state.storage`, one DO instance per session |
156 | Any platform, in-memory (tests) | `Persistence.layerBackingMemory` — lost on restart |
157 | Anything else (KV, R2, DynamoDB, …) | Implement `BackingPersistence` against your store |
158
159 The `Chat.Persistence` API and handler code never change — only the
160 backing layer does.
161
162 ## Cloudflare Workers AI binding
163
164 For Workers AI models on Cloudflare, the shortest path is the native
165 [Workers AI binding](/cloudflare/ai/workers-ai): no API key, no HTTP
166 client layer, no resource to declare. `yield*
167 Cloudflare.Workers.AI()` in the Construction phase attaches the binding, and
168 `ai.model({...})` builds the `LanguageModel` Layer directly on top of
169 it:
170
171 ```typescript
172 Effect.gen(function* () {
173 const ai = yield* Cloudflare.Workers.AI();
174
175 const languageModel = ai.model({
176 model: "@cf/meta/llama-3.3-70b-instruct-fp8-fast",
177 parameters: { temperature: 0.7, maxTokens: 1024 },
178 });
179
180 return {
181 fetch: Effect.gen(function* () {
182 const response = yield* LanguageModel.generateText({
183 prompt: "Say hello.",
184 }).pipe(Effect.orDie);
185 return yield* HttpServerResponse.json({ text: response.text });
186 }).pipe(Effect.provide(languageModel)),
187 };
188 }).pipe(Effect.provide(Cloudflare.Workers.AIBinding));
189 ```
190
191 The [Workers AI guide](/cloudflare/ai/workers-ai) covers the rest of
192 the binding surface — raw `ai.run(...)` inference, listing models,
193 and declaring the binding on a plain async Worker's `env`.
194
195 ## Cloudflare AI Gateway
196
197 To route Workers AI through an **AI Gateway** instead — caching, rate
198 limiting, retries, and a unified request log — `Cloudflare.AI.Gateway`
199 declares the gateway as a resource and
200 `Cloudflare.AI.QueryGateway(Gateway)` returns a typed client whose
201 `.model({...})` builds the same `LanguageModel` Layer shape, proxied
202 through the gateway.
203
204 Bind the gateway in the Construction phase and call `aiGateway.model({...})`
205 with a Workers AI model id. It returns a `LanguageModel` Layer
206 directly — no API key, no `Layer.unwrap`, since the gateway binding
207 handles auth and the URL:
208
209 ```typescript
210 const aiGateway = yield* Cloudflare.AI.QueryGateway(Gateway);
211
212 const languageModel = aiGateway.model({
213 model: "@cf/meta/llama-3.1-8b-instruct",
214 parameters: { temperature: 0.7, maxTokens: 1024 },
215 });
216 ```
217
218 That `languageModel` Layer slots into the same `Effect.provide(...)`
219 spot as any other provider — the rest of your handler doesn't
220 change. Provide `Cloudflare.AI.QueryGatewayBinding` once at the
221 bottom of the constructor's layer chain so the binding resolves at runtime:
222
223 ```typescript
224 Effect.gen(function* () {
225 const aiGateway = yield* Cloudflare.AI.QueryGateway(Gateway);
226
227 const languageModel = aiGateway.model({
228 model: "@cf/meta/llama-3.1-8b-instruct",
229 parameters: { temperature: 0.7, maxTokens: 1024 },
230 });
231
232 return {
233 fetch: Effect.gen(function* () {
234 const response = yield* LanguageModel.generateText({
235 prompt: "Say hello.",
236 }).pipe(Effect.orDie);
237 return yield* HttpServerResponse.json({ text: response.text });
238 }).pipe(Effect.provide(languageModel)),
239 };
240 }).pipe(Effect.provide(Cloudflare.AI.QueryGatewayBinding));
241 ```
242
243 The [AI Gateway tutorial](/cloudflare/ai/ai-gateway) walks
244 through it end to end — declaring the resource, streaming with
245 `streamText`, and tuning caching, rate limits, and DLP.
246
247 ## Picking a provider
248
249 Each upstream provider is its own `@effect/ai-*` package. The
250 shape — `*Client.layer({ apiKey })` + `*LanguageModel.layer({
251 model })` — is the same.
252
253 - [`@effect/ai-openai`](https://www.npmjs.com/package/@effect/ai-openai)
254 - [`@effect/ai-anthropic`](https://www.npmjs.com/package/@effect/ai-anthropic)
255 - [`@effect/ai-google`](https://www.npmjs.com/package/@effect/ai-google)
256 - [`@effect/ai-amazon-bedrock`](https://www.npmjs.com/package/@effect/ai-amazon-bedrock)
257 - `Cloudflare.Workers.AI` (the native Workers AI binding, via Alchemy itself)
258 - `Cloudflare.AI.Gateway` (Workers AI behind an AI Gateway, via Alchemy itself)
259 - [`AWS.Bedrock.LanguageModel`](/aws/ai/bedrock) (Amazon Bedrock over the
260 Converse API with IAM-scoped bindings, via Alchemy itself)
261
262 ## Where to go next
263
264 - [Effect AI documentation](https://effect.website/docs/ai/introduction)
265 — the API surface (`generateText`, `streamText`, `Toolkit`,
266 structured outputs, embeddings).
267 - [Run Workers AI models](/cloudflare/ai/workers-ai) — the native
268 Workers AI binding, with or without the `LanguageModel` layer.
269 - [Add an AI Gateway](/cloudflare/ai/ai-gateway) — wire
270 Workers AI behind a Cloudflare AI Gateway with caching and
271 streaming.
272 - [Secrets & env](/cloudflare/security/secrets-env) — how `Config.Redacted`
273 feeds the upstream providers' API keys at deploy and runtime.