The best way to control Claude Code, Codex and more from any computer or phone

main
mdx 225 lines 7.84 KB
Raw
1 ---
2 title: Run Workers AI models
3 description: Bind Workers AI into your Worker with Cloudflare.Workers.AI — run inference and list models directly, or turn the binding into a typed Effect LanguageModel, no gateway required.
4 sidebar:
5 order: 4
6 ---
7
8 **Workers AI** runs open-weight models (Llama, Mistral, Whisper,
9 embeddings, …) on Cloudflare's GPU fleet, callable from any Worker
10 through the native `env.AI` binding. There is no resource to declare
11 and nothing to provision — the service is account-level, so the only
12 Alchemy piece is the binding itself: `Cloudflare.Workers.AI`.
13
14 This is the plain `{ type: "ai" }` Worker binding, the same one you
15 would declare in `wrangler.json`. If you want requests routed through
16 an **AI Gateway** — caching, rate limiting, retries, a request log —
17 use [`Cloudflare.AI.Gateway`](/cloudflare/ai/ai-gateway) instead; the
18 binding here talks to Workers AI directly.
19
20 ## Effect-style Worker
21
22 `yield* Cloudflare.Workers.AI()` inside a Worker's Construction phase does two
23 things: it attaches the `ai` binding to the Worker at deploy time, and
24 at runtime it resolves to an Effect-native client whose `run` and
25 `models` operations return Effects with a typed `WorkersAIError`
26 channel. Provide `Cloudflare.Workers.AIBinding` once at the bottom of
27 the constructor's layer chain, the same way every other Worker binding works:
28
29 ```typescript
30 // src/AiWorker.ts
31 import * as Cloudflare from "@alchemy.run/cloudflare";
32 import * as Effect from "effect/Effect";
33 import { HttpServerRequest } from "effect/http/HttpServerRequest";
34 import * as HttpServerResponse from "effect/http/HttpServerResponse";
35
36 const MODEL = "@cf/meta/llama-3.3-70b-instruct-fp8-fast";
37
38 export default class AiWorker extends Cloudflare.Worker<AiWorker>()(
39 "AiWorker",
40 { main: import.meta.url },
41 Effect.gen(function* () {
42 const ai = yield* Cloudflare.Workers.AI();
43
44 return {
45 fetch: Effect.gen(function* () {
46 const request = yield* HttpServerRequest;
47 const url = new URL(request.url, "http://worker");
48
49 if (url.pathname === "/run") {
50 const result = yield* ai
51 .run(MODEL, {
52 messages: [{ role: "user", content: "Say hello in one sentence." }],
53 max_tokens: 128,
54 })
55 .pipe(Effect.orDie);
56 return yield* HttpServerResponse.json(result);
57 }
58
59 if (url.pathname === "/models") {
60 const models = yield* ai
61 .models({ search: "llama-3.3" })
62 .pipe(Effect.orDie);
63 return yield* HttpServerResponse.json({
64 names: models.map((m) => m.name),
65 });
66 }
67
68 return HttpServerResponse.text("Not Found", { status: 404 });
69 }),
70 };
71 }).pipe(Effect.provide(Cloudflare.Workers.AIBinding)),
72 ) {}
73 ```
74
75 `ai.run` is typed by the model catalog from
76 `@cloudflare/workers-types`: the model id narrows the `inputs` and
77 output types, so `@cf/meta/...` chat models take `messages` /
78 `prompt` while embedding models take `text`, checked at compile time.
79 `Effect.orDie` collapses `WorkersAIError` to a defect so a model
80 failure surfaces as a 500 — for typed handling use
81 `Effect.catchTag("WorkersAIError", …)` instead.
82
83 ## Build a `LanguageModel` layer
84
85 `ai.model({...})` turns the binding into an
86 [Effect AI](/cloudflare/ai/effect-ai) `LanguageModel` Layer — the
87 provider-agnostic service behind `generateText`, `streamText`,
88 `Toolkit`, and structured outputs. It is the same adapter that AI
89 Gateway's `QueryGateway` uses, minus the gateway routing, so no API
90 key and no HTTP client layer are involved:
91
92 ```typescript
93 import { LanguageModel } from "effect/ai";
94
95 Effect.gen(function* () {
96 const ai = yield* Cloudflare.Workers.AI();
97
98 const languageModel = ai.model({
99 model: "@cf/meta/llama-3.3-70b-instruct-fp8-fast",
100 parameters: { temperature: 0.7, maxTokens: 1024 },
101 });
102
103 return {
104 fetch: Effect.gen(function* () {
105 const response = yield* LanguageModel.generateText({
106 prompt: "Write a haiku about Effect.",
107 }).pipe(Effect.orDie);
108 return yield* HttpServerResponse.json({
109 text: response.text,
110 usage: {
111 inputTokens: response.usage.inputTokens.total,
112 outputTokens: response.usage.outputTokens.total,
113 },
114 });
115 }).pipe(Effect.provide(languageModel)),
116 };
117 }).pipe(Effect.provide(Cloudflare.Workers.AIBinding));
118 ```
119
120 Streaming works the same way as any other provider —
121 `LanguageModel.streamText` returns a stream of typed response parts.
122 Use `Stream.provide(languageModel)` (the stream-aware equivalent of
123 `Effect.provide`) so the model stays available for the lifetime of
124 the stream:
125
126 ```typescript
127 import * as Stream from "effect/Stream";
128 import * as Sse from "effect/encoding/Sse";
129
130 const stream = LanguageModel.streamText({ prompt }).pipe(
131 Stream.provide(languageModel),
132 Sse.encode,
133 );
134
135 return HttpServerResponse.stream(stream, {
136 headers: { "content-type": "text/event-stream" },
137 });
138 ```
139
140 Because the Layer satisfies Effect's standard `LanguageModel`
141 service, swapping Workers AI for OpenAI, Anthropic, or a
142 gateway-fronted model later is a layer-level change — the handler
143 code doesn't move. See [Effect AI](/cloudflare/ai/effect-ai) for the
144 full pattern, including chat persistence.
145
146 ## Read the neurons a response billed
147
148 Workers AI bills in neurons and reports the count with every
149 response. The adapter carries it on the `finish` part, and
150 `Cloudflare.AI.finishNeurons` reads it back as `number | undefined`:
151
152 ```typescript
153 const response = yield* LanguageModel.generateText({ prompt });
154 const finish = response.content.find((part) => part.type === "finish");
155 const neurons = finish ? Cloudflare.AI.finishNeurons(finish) : undefined;
156 ```
157
158 Streams carry it on their final `finish` part too. It is `undefined`
159 for models that don't report neurons.
160
161 ## Async Workers: bind via `env`
162
163 A plain async Worker (no Effect runtime) declares the binding on
164 `env` instead.
165 `InferEnv` maps it to the native `Ai` runtime handle — exactly what
166 `wrangler` would give you:
167
168 ```typescript
169 // src/AiAsyncWorker.ts
170 import * as Cloudflare from "@alchemy.run/cloudflare";
171
172 export const AiAsyncWorker = Cloudflare.Worker("AiAsyncWorker", {
173 main: "./src/handler.ts",
174 env: {
175 AI: Cloudflare.Workers.AI(),
176 },
177 });
178
179 export type AiAsyncWorkerEnv = Cloudflare.InferEnv<typeof AiAsyncWorker>;
180 // { AI: Ai }
181 ```
182
183 ```typescript
184 // src/handler.ts
185 import type { AiAsyncWorkerEnv } from "./AiAsyncWorker.ts";
186
187 export default {
188 async fetch(request: Request, env: AiAsyncWorkerEnv): Promise<Response> {
189 const result = await env.AI.run("@cf/meta/llama-3.3-70b-instruct-fp8-fast", {
190 messages: [{ role: "user", content: "Say hello in one sentence." }],
191 max_tokens: 128,
192 });
193 return Response.json(result);
194 },
195 };
196 ```
197
198 `Cloudflare.Workers.AI()` takes an optional binding name (the `env`
199 key it resolves to) and defaults to `"AI"`.
200
201 ## When to add an AI Gateway
202
203 The plain binding is the shortest path to inference, but every
204 request goes straight to Workers AI. Put an
205 [AI Gateway](/cloudflare/ai/ai-gateway) in front when you want:
206
207 - **Caching** — identical prompts served from cache in milliseconds.
208 - **Rate limiting and retries** — protect quota, smooth over blips.
209 - **A request log** — every prompt, completion, latency, and token
210 count in one dashboard.
211 - **Other providers** — one endpoint fronting OpenAI, Anthropic,
212 Bedrock, … alongside Workers AI.
213
214 `aiGateway.model({...})` and `ai.model({...})` produce the same
215 `LanguageModel` Layer shape, so moving between them is a one-line
216 change in the Construction phase.
217
218 ## Where to go next
219
220 - [Effect AI](/cloudflare/ai/effect-ai) — the `LanguageModel` /
221 `Chat` pattern this binding plugs into.
222 - [Add an AI Gateway](/cloudflare/ai/ai-gateway) — caching, rate
223 limiting, and logs in front of the same models.
224 - [API reference: Cloudflare.Workers.AI](/providers/cloudflare/workers/ai)
225 — the full binding surface (`run`, `models`, `model`, `raw`).