@devries-cameron20 / t3code Mirror · read-only
Fork 0

The best way to control Claude Code, Codex and more from any computer or phone

Mirrored from https://github.com/Camo67/t3code.git

Synced Nov 21, 04:45 UTC

main
mdx 240 lines 8.22 KB
Raw
1 ---
2 title: Pipelines
3 description: Ingest events into R2 with Cloudflare Pipelines — a typed stream, a SQL pipeline, and a sink that writes JSON or Parquet files, or Apache Iceberg tables.
4 ---
5
6 **Cloudflare Pipelines** ingests events into R2. Producers send JSON events to a
7 **stream**. A **pipeline** filters or reshapes them with SQL and writes them to
8 a **sink**: plain JSON or Parquet files in a bucket, or rows in an Apache
9 Iceberg table.
10
11 Pipelines is part of **Basin**, Cloudflare's analytics stack on R2, and its
12 resources are also exported as `Cloudflare.Basin.*`. Reach for Pipelines when
13 events should land as files or tables you query later. Reach for
14 [K2](/cloudflare/messaging/k2) or [Queues](/cloudflare/messaging/queues) when
15 your own code consumes the events.
16
17 By the end of this page, a Worker sends page views to a stream, and a pipeline
18 writes them to an Iceberg table.
19
20 ## Create a typed stream
21
22 Declare the stream with an Effect Schema. Alchemy converts it into the stream's
23 field list, which Cloudflare enforces.
24
25 ```typescript
26 // src/Events.ts
27 import * as Cloudflare from "@alchemy.run/cloudflare";
28 import * as Schema from "effect/Schema";
29
30 export const PageView = Schema.Struct({
31 userId: Schema.String,
32 path: Schema.String,
33 at: Schema.Date,
34 tags: Schema.optional(Schema.Array(Schema.String)),
35 });
36
37 export const PageViews = Cloudflare.Pipelines.Stream("PageViews", {
38 schema: PageView,
39 });
40 ```
41
42 Arrays become `list` fields, nested structs become `struct` fields, and
43 `Schema.optional` fields are not required. A schema with no stream
44 representation, such as a union, fails when the program is built.
45
46 A stream's schema can't change after it is created. Changing it replaces the
47 stream, which drops events it hasn't processed yet and deletes the pipelines
48 that read from it. Writing the same schema as an Effect Schema or as an
49 explicit `{ fields }` list is the same stream and causes no change.
50
51 ## Send events from a Worker
52
53 `Cloudflare.Pipelines.WriteStream` is typed by the stream's schema. The client
54 encodes each value with the schema before sending it.
55
56 ```typescript
57 // src/Api.ts
58 import * as Cloudflare from "@alchemy.run/cloudflare";
59 import * as Effect from "effect/Effect";
60 import { HttpServerRequest } from "effect/http/HttpServerRequest";
61 import * as HttpServerResponse from "effect/http/HttpServerResponse";
62 import { PageViews } from "./Events.ts";
63
64 export default class Api extends Cloudflare.Worker<Api>()(
65 "Api",
66 { main: import.meta.url },
67 Effect.gen(function* () {
68 const views = yield* Cloudflare.Pipelines.WriteStream(PageViews);
69
70 return {
71 fetch: Effect.gen(function* () {
72 const request = yield* HttpServerRequest;
73 yield* views.send([{ userId: "anon", path: request.url, at: new Date() }]);
74 return HttpServerResponse.text("ok");
75 }),
76 };
77 }).pipe(Effect.provide(Cloudflare.Pipelines.WriteStreamBinding)),
78 ) {}
79 ```
80
81 Cloudflare accepts events that don't match a stream's schema and drops them
82 later during processing, so the client-side encoding is the only validation.
83
84 To drain an Effect `Stream` into the pipeline, use
85 `Cloudflare.Pipelines.StreamSink`. It packs events into requests of up to 5 MB:
86
87 ```typescript
88 const sink = yield* Cloudflare.Pipelines.StreamSink(PageViews);
89 yield* Stream.fromIterable(events).pipe(Stream.run(sink));
90 ```
91
92 Provide `Cloudflare.Pipelines.StreamSinkBinding` alongside `WriteStreamBinding`.
93
94 ## Send events from outside a Worker
95
96 The producer code stays the same on every host. Only the layer changes:
97
98 - `WriteStreamBinding` and `StreamSinkBinding` run in Cloudflare Workers and
99 send through the native `pipelines` binding.
100 - `WriteStreamHttp` and `StreamSinkHttp` run on any host. They send to the
101 stream's HTTP endpoint with a scoped "Pipelines Send" token that Alchemy mints
102 at deploy.
103 - `WriteStreamLocal` and `StreamSinkLocal` run in scripts and Actions. They
104 send to the stream's HTTP endpoint with your current Cloudflare credentials.
105
106 The HTTP and local layers need the stream's HTTP endpoint turned on.
107
108 ## Turn on the HTTP endpoint
109
110 A stream gets an HTTP ingest endpoint only when you ask for one:
111
112 ```typescript
113 Cloudflare.Pipelines.Stream("PageViews", { http: true }); // authenticated
114 Cloudflare.Pipelines.Stream("PageViews", {
115 http: { enabled: true, authentication: false }, // public, e.g. for browsers
116 });
117 ```
118
119 :::caution
120 Earlier versions of Alchemy left a stream declared without `http` with
121 Cloudflare's default, a public endpoint that anyone can write to. The next
122 deploy now closes that endpoint in place, without replacing the stream. If
123 external producers post to it, keep it public with
124 `http: { enabled: true, authentication: false }`.
125 :::
126
127 ## Write events to an Iceberg table
128
129 A catalog sink writes the pipeline's output to an
130 [Iceberg table](/cloudflare/data/iceberg-tables). It needs a bucket with the
131 catalog enabled. Declare them in your stack's Effect, and pass the catalog
132 resource to the sink so it is created after the catalog is enabled:
133
134 ```typescript
135 // alchemy.run.ts
136 import * as Alchemy from "alchemy";
137 import * as Cloudflare from "@alchemy.run/cloudflare";
138 import * as Config from "effect/Config";
139 import * as Effect from "effect/Effect";
140 import { PageViews } from "./src/Events.ts";
141
142 export default Alchemy.Stack(
143 "Analytics",
144 {
145 providers: Cloudflare.providers(),
146 state: Cloudflare.state(),
147 },
148 Effect.gen(function* () {
149 const lake = yield* Cloudflare.R2.Bucket("Lake");
150 const catalog = yield* Cloudflare.R2.DataCatalog("Catalog", {
151 bucket: lake,
152 });
153 const views = yield* PageViews;
154
155 const table = yield* Cloudflare.Pipelines.Sink("PageViewTable", {
156 type: "basin_catalog",
157 catalog,
158 table: { namespace: "web", name: "page_views" },
159 token: yield* Config.Redacted("CATALOG_TOKEN"),
160 });
161 }),
162 );
163 ```
164
165 The sink creates the namespace and table on its first write. The `token` is a
166 Cloudflare API token with access to R2 and the catalog. Rotating it updates the
167 sink without replacing it.
168
169 :::caution
170 A catalog sink can't be created for a table that already exists. Changing the
171 sink's other settings replaces the sink, and the replacement fails once the
172 table exists. Choose the table and format before the first deploy.
173 :::
174
175 ## Connect the stream and sink with SQL
176
177 Add the pipeline in the same Effect:
178
179 ```diff lang="typescript"
180 // alchemy.run.ts
181 +import * as Output from "alchemy/Output";
182
183 const table = yield* Cloudflare.Pipelines.Sink("PageViewTable", { /* ... */ });
184 +
185 + yield* Cloudflare.Pipelines.Pipeline("PageViewEtl", {
186 + sql: Output.interpolate`
187 + INSERT INTO ${table.name}
188 + SELECT userId, path, at FROM ${views.name}
189 + WHERE path NOT LIKE '/health%'`,
190 + });
191 ```
192
193 Alchemy validates the SQL with Cloudflare during plan, so a bad statement fails
194 before anything is replaced. Reformatting the SQL is not a change.
195
196 ## Write files instead of a table
197
198 An `r2` sink writes events to a bucket as JSON or Parquet files, without a
199 catalog:
200
201 ```typescript
202 const archive = yield* Cloudflare.Pipelines.Sink("EventArchive", {
203 type: "r2",
204 config: {
205 bucket: lake.bucketName,
206 credentials: {
207 accessKeyId: yield* Config.Redacted("R2_ACCESS_KEY_ID"),
208 secretAccessKey: yield* Config.Redacted("R2_SECRET_ACCESS_KEY"),
209 },
210 path: "events",
211 rollingPolicy: { intervalSeconds: 60 },
212 },
213 format: { type: "parquet", compression: "zstd" },
214 });
215 ```
216
217 `rollingPolicy` decides when the sink closes a file and starts the next one:
218 after `intervalSeconds`, after `inactivitySeconds` without new events, or when
219 the file reaches `fileSizeBytes`. Point a pipeline at it the same way, with `${archive.name}` in
220 the SQL. Unlike a catalog sink, an `r2` sink can be replaced freely.
221
222 ## React to new files
223
224 Pipelines has no event source of its own, because its consumer is the sink. To
225 process the files a sink writes, listen for object events on the bucket with
226 `Cloudflare.R2.BucketEventNotification`.
227
228 ## Where next
229
230 Related:
231
232 - [Iceberg tables](/cloudflare/data/iceberg-tables): manage the tables a
233 catalog sink writes, and query them.
234 - [R2](/cloudflare/data/r2): the bucket a sink writes to.
235 - [K2 streams](/cloudflare/messaging/k2): a durable log your own code consumes.
236
237 Reference:
238
239 - [Pipelines API reference](/providers/cloudflare/pipelines): `Stream`, `Sink`,
240 `Pipeline`, `WriteStream`, and `StreamSink`.