@devries-cameron20 / t3code Mirror · read-only
Fork 0

The best way to control Claude Code, Codex and more from any computer or phone

Mirrored from https://github.com/Camo67/t3code.git

Synced Nov 21, 06:00 UTC

main
mdx 281 lines 9.49 KB
Raw
1 ---
2 title: Add an AI Gateway
3 description: Wire an AI Gateway into your Worker, turn it into a typed Effect LanguageModel, and run generations and streams through Workers AI with caching, rate limiting, and logs.
4 sidebar:
5 order: 6
6 ---
7
8 You've now wired Durable Objects, hibernatable WebSockets, a
9 container, and a Workflow into your Worker. The last piece in the
10 Cloudflare track is an **AI Gateway** — a stable account-scoped
11 endpoint that fronts every model provider (Workers AI, OpenAI,
12 Anthropic, Bedrock, …) and gives you caching, rate limiting, retries,
13 DLP, and a single dashboard of every request, token, and cost.
14
15 In Alchemy the gateway is a single resource. Once you've declared and
16 bound it, `.model({...})` returns an Effect `LanguageModel.LanguageModel`
17 Layer — and from there you're using the same `generateText` /
18 `streamText` APIs you'd use against any other provider.
19
20 Don't need caching, rate limiting, or the request log? The plain
21 [Workers AI binding](/cloudflare/ai/workers-ai)
22 (`Cloudflare.Workers.AI`) calls the same models directly, with no
23 gateway resource — and its `ai.model({...})` produces the same
24 `LanguageModel` Layer, so you can start there and add the gateway
25 later without touching handler code.
26
27 ## Declare the gateway
28
29 Create `src/AiGateway.ts` with a single resource definition. The
30 two flags below enable response caching (60-second TTL) and request
31 logging — every prompt, completion, latency, and token count will
32 show up in the AI Gateway dashboard.
33
34 ```typescript
35 // src/AiGateway.ts
36 import * as Cloudflare from "@alchemy.run/cloudflare";
37
38 export const Gateway = Cloudflare.AI.Gateway("Gateway", {
39 cacheTtl: 60,
40 collectLogs: true,
41 });
42 ```
43
44 Every prop is optional, but explicit defaults make the intent
45 visible. We'll tune more knobs at the end of the tutorial.
46
47 ## Add it to the stack
48
49 ```diff lang="typescript"
50 // alchemy.run.ts
51 +import { Gateway } from "./src/AiGateway.ts";
52 import Api from "./src/Api.ts";
53
54 export default Alchemy.Stack(
55 "CloudflareWorkerExample",
56 { providers: Cloudflare.providers(), state: Cloudflare.state() },
57 Effect.gen(function* () {
58 const api = yield* Api;
59 + const gateway = yield* Gateway;
60
61 return {
62 url: api.url.as<string>(),
63 + gatewayId: gateway.gatewayId,
64 };
65 }),
66 );
67 ```
68
69 `yield* Gateway` registers the resource so it gets created/updated on
70 the next deploy. `gateway.gatewayId` is exposed as a stack output so
71 you can find it in the dashboard.
72
73 ## Bind the gateway into the Worker
74
75 `Cloudflare.AI.QueryGateway(Gateway)` returns a typed client whose
76 methods are wrapped in Effect — `run` for raw inference, `getUrl` for
77 the gateway endpoint, `getLog`/`patchLog` for the request log, and
78 `model({...})` for building a `LanguageModel` layer.
79
80 ```diff lang="typescript"
81 // src/Api.ts
82 import * as Cloudflare from "@alchemy.run/cloudflare";
83 import * as Effect from "effect/Effect";
84 +import { Gateway } from "./AiGateway.ts";
85
86 export default class Api extends Cloudflare.Worker<Api>()(
87 "Api",
88 { main: import.meta.url },
89 Effect.gen(function* () {
90 + const aiGateway = yield* Cloudflare.AI.QueryGateway(Gateway);
91
92 return {
93 fetch: Effect.gen(function* () {
94 // …existing routes
95 }),
96 };
97 - }),
98 + }).pipe(Effect.provide(Cloudflare.AI.QueryGatewayBinding)),
99 ) {}
100 ```
101
102 `Cloudflare.AI.QueryGatewayBinding` is the runtime side of the
103 binding. Provide it once at the bottom of the constructor's layer chain and
104 every `QueryGateway(...)` further up will resolve.
105
106 ## Build a `LanguageModel` layer
107
108 Call `aiGateway.model({...})` with a Workers AI model id and
109 parameters. The result is a `Layer<LanguageModel.LanguageModel, …>`
110 that satisfies Effect's standard AI service.
111
112 ```diff lang="typescript"
113 Effect.gen(function* () {
114 const aiGateway = yield* Cloudflare.AI.QueryGateway(Gateway);
115 +
116 + const languageModel = aiGateway.model({
117 + model: "@cf/meta/llama-3.3-70b-instruct-fp8-fast",
118 + parameters: { temperature: 0.7, maxTokens: 1024 },
119 + });
120
121 return {
122 fetch: Effect.gen(function* () {
123 // …
124 }),
125 };
126 })
127 ```
128
129 `parameters` is the per-call default; any individual call can still
130 override it. The Construction phase is the right place to build this layer —
131 construction is pure and the binding factory only exists here.
132
133 ## Generate text on `/generate`
134
135 `LanguageModel.generateText` returns a typed response with `text`,
136 `finishReason`, structured token `usage`, and any `toolCalls`. Provide
137 the `languageModel` layer to the handler and call it like any other
138 Effect.
139
140 ```diff lang="typescript"
141 +import { LanguageModel } from "effect/ai";
142 import { HttpServerRequest } from "effect/http/HttpServerRequest";
143 import * as HttpServerResponse from "effect/http/HttpServerResponse";
144
145 return {
146 fetch: Effect.gen(function* () {
147 const request = yield* HttpServerRequest;
148 + const url = new URL(request.url, "http://api");
149
150 + if (url.pathname === "/generate" && request.method === "POST") {
151 + const body = (yield* request.json) as { prompt?: string };
152 + const prompt = body.prompt?.trim() ?? "Say hello in one sentence.";
153 +
154 + const response = yield* LanguageModel.generateText({ prompt }).pipe(
155 + Effect.orDie,
156 + );
157 +
158 + return yield* HttpServerResponse.json({
159 + text: response.text,
160 + usage: {
161 + inputTokens: response.usage.inputTokens.total,
162 + outputTokens: response.usage.outputTokens.total,
163 + },
164 + });
165 + }
166
167 return HttpServerResponse.text("Not Found", { status: 404 });
168 - }),
169 + }).pipe(Effect.provide(languageModel)),
170 };
171 ```
172
173 `Effect.provide(languageModel)` makes `LanguageModel.LanguageModel`
174 available to every handler in `fetch`. `Effect.orDie` collapses
175 `AiError` to a defect so a model failure surfaces as a 500 — if you
176 need typed handling instead, `Effect.catchTag("AiError", …)` works.
177
178 ## Try it
179
180 ```sh
181 bun alchemy deploy
182 curl -X POST "$(bun alchemy stack output url)/generate" \
183 -H "content-type: application/json" \
184 -d '{"prompt":"Write a haiku about Effect"}'
185 ```
186
187 The first call takes ~1–2 seconds. Send the **exact same** prompt
188 again and it returns in milliseconds — that's the `cacheTtl: 60`
189 config doing its job. Open the Cloudflare dashboard → **AI** →
190 **AI Gateway** → your gateway and you'll see both requests, with the
191 second flagged as a cache hit, plus latency and token usage on every
192 entry.
193
194 ## Stream tokens on `/stream`
195
196 For chat-style UIs you want tokens to arrive as the model produces
197 them, not in one big response. `LanguageModel.streamText` returns an
198 `Effect.Stream` of typed response parts — `text-delta`, `tool-call`,
199 `finish`, and so on. Pipe it through `Sse.encode` and
200 `HttpServerResponse.stream` to get a server-sent-event stream that
201 flushes through the Worker → edge → client without buffering.
202
203 ```diff lang="typescript"
204 +import * as Stream from "effect/Stream";
205 +import * as Sse from "effect/encoding/Sse";
206
207 + if (url.pathname === "/stream" && request.method === "POST") {
208 + const body = (yield* request.json) as { prompt?: string };
209 + const prompt = body.prompt?.trim() ?? "Tell me a haiku about Effect.";
210 +
211 + const stream = LanguageModel.streamText({ prompt }).pipe(
212 + Stream.provide(languageModel),
213 + Sse.encode,
214 + );
215 +
216 + return HttpServerResponse.stream(stream, {
217 + headers: {
218 + "content-type": "text/event-stream",
219 + "cache-control": "no-cache",
220 + "x-accel-buffering": "no",
221 + },
222 + });
223 + }
224 ```
225
226 `Stream.provide(languageModel)` is the stream-aware equivalent of
227 `Effect.provide` — the `LanguageModel` needs to be available for the
228 entire lifetime of the stream, not just the initial setup.
229
230 ```sh
231 curl -N -X POST "$(bun alchemy stack output url)/stream" \
232 -H "content-type: application/json" \
233 -d '{"prompt":"Write a haiku about Effect, slowly."}'
234 ```
235
236 `-N` disables curl's response buffering so you see each SSE event as
237 soon as the Worker flushes it.
238
239 ## Tune caching, rate limits, and DLP
240
241 Every prop on `Cloudflare.AI.Gateway` maps to an update API call — no
242 replacement, no downtime. A production-grade config might look like:
243
244 ```typescript
245 export const Gateway = Cloudflare.AI.Gateway("Gateway", {
246 id: "prod-gateway",
247 cacheTtl: 300,
248 cacheInvalidateOnUpdate: true,
249 rateLimitingInterval: 60,
250 rateLimitingLimit: 100,
251 rateLimitingTechnique: "sliding",
252 collectLogs: true,
253 logManagement: 100_000,
254 logManagementStrategy: "DELETE_OLDEST",
255 authentication: true,
256 });
257 ```
258
259 Bumping `cacheTtl`, `rateLimitingLimit`, or toggling `authentication`
260 is a single `bun alchemy deploy` away — the diff updates the gateway
261 in place.
262
263 ## What you have now
264
265 - A `Cloudflare.AI.Gateway` resource with caching, logging, and (when
266 you want them) rate limiting and auth.
267 - An Effect `LanguageModel.LanguageModel` Layer that proxies Workers
268 AI through that gateway.
269 - `/generate` and `/stream` routes that use the standard
270 `LanguageModel.generateText` and `streamText` APIs — provider-
271 agnostic, so swapping Workers AI for OpenAI or Anthropic later is
272 a layer-level change, not a code-level one.
273
274 The natural next step is to give the model **memory** — wrapping the
275 same `LanguageModel` with `Chat.Service` and storing the conversation
276 in a Durable Object's `state.storage`, so every session is resumable
277 across requests, restarts, and hibernation.
278
279 For the wider API surface — `generateObject`, `Toolkit`, structured
280 outputs, and how the same `LanguageModel` composes inside an HTTP
281 API or RPC handler — see the [Effect AI guide](/cloudflare/ai/effect-ai).