The best way to control Claude Code, Codex and more from any computer or phone
Mirrored from https://github.com/Camo67/t3code.git
Synced Nov 21, 04:45 UTC
| 1 | --- |
| 2 | title: Pipelines |
| 3 | description: Ingest events into R2 with Cloudflare Pipelines — a typed stream, a SQL pipeline, and a sink that writes JSON or Parquet files, or Apache Iceberg tables. |
| 4 | --- |
| 5 | |
| 6 | **Cloudflare Pipelines** ingests events into R2. Producers send JSON events to a |
| 7 | **stream**. A **pipeline** filters or reshapes them with SQL and writes them to |
| 8 | a **sink**: plain JSON or Parquet files in a bucket, or rows in an Apache |
| 9 | Iceberg table. |
| 10 | |
| 11 | Pipelines is part of **Basin**, Cloudflare's analytics stack on R2, and its |
| 12 | resources are also exported as `Cloudflare.Basin.*`. Reach for Pipelines when |
| 13 | events should land as files or tables you query later. Reach for |
| 14 | [K2](/cloudflare/messaging/k2) or [Queues](/cloudflare/messaging/queues) when |
| 15 | your own code consumes the events. |
| 16 | |
| 17 | By the end of this page, a Worker sends page views to a stream, and a pipeline |
| 18 | writes them to an Iceberg table. |
| 19 | |
| 20 | ## Create a typed stream |
| 21 | |
| 22 | Declare the stream with an Effect Schema. Alchemy converts it into the stream's |
| 23 | field list, which Cloudflare enforces. |
| 24 | |
| 25 | ```typescript |
| 26 | // src/Events.ts |
| 27 | import * as Cloudflare from "@alchemy.run/cloudflare"; |
| 28 | import * as Schema from "effect/Schema"; |
| 29 | |
| 30 | export const PageView = Schema.Struct({ |
| 31 | userId: Schema.String, |
| 32 | path: Schema.String, |
| 33 | at: Schema.Date, |
| 34 | tags: Schema.optional(Schema.Array(Schema.String)), |
| 35 | }); |
| 36 | |
| 37 | export const PageViews = Cloudflare.Pipelines.Stream("PageViews", { |
| 38 | schema: PageView, |
| 39 | }); |
| 40 | ``` |
| 41 | |
| 42 | Arrays become `list` fields, nested structs become `struct` fields, and |
| 43 | `Schema.optional` fields are not required. A schema with no stream |
| 44 | representation, such as a union, fails when the program is built. |
| 45 | |
| 46 | A stream's schema can't change after it is created. Changing it replaces the |
| 47 | stream, which drops events it hasn't processed yet and deletes the pipelines |
| 48 | that read from it. Writing the same schema as an Effect Schema or as an |
| 49 | explicit `{ fields }` list is the same stream and causes no change. |
| 50 | |
| 51 | ## Send events from a Worker |
| 52 | |
| 53 | `Cloudflare.Pipelines.WriteStream` is typed by the stream's schema. The client |
| 54 | encodes each value with the schema before sending it. |
| 55 | |
| 56 | ```typescript |
| 57 | // src/Api.ts |
| 58 | import * as Cloudflare from "@alchemy.run/cloudflare"; |
| 59 | import * as Effect from "effect/Effect"; |
| 60 | import { HttpServerRequest } from "effect/http/HttpServerRequest"; |
| 61 | import * as HttpServerResponse from "effect/http/HttpServerResponse"; |
| 62 | import { PageViews } from "./Events.ts"; |
| 63 | |
| 64 | export default class Api extends Cloudflare.Worker<Api>()( |
| 65 | "Api", |
| 66 | { main: import.meta.url }, |
| 67 | Effect.gen(function* () { |
| 68 | const views = yield* Cloudflare.Pipelines.WriteStream(PageViews); |
| 69 | |
| 70 | return { |
| 71 | fetch: Effect.gen(function* () { |
| 72 | const request = yield* HttpServerRequest; |
| 73 | yield* views.send([{ userId: "anon", path: request.url, at: new Date() }]); |
| 74 | return HttpServerResponse.text("ok"); |
| 75 | }), |
| 76 | }; |
| 77 | }).pipe(Effect.provide(Cloudflare.Pipelines.WriteStreamBinding)), |
| 78 | ) {} |
| 79 | ``` |
| 80 | |
| 81 | Cloudflare accepts events that don't match a stream's schema and drops them |
| 82 | later during processing, so the client-side encoding is the only validation. |
| 83 | |
| 84 | To drain an Effect `Stream` into the pipeline, use |
| 85 | `Cloudflare.Pipelines.StreamSink`. It packs events into requests of up to 5 MB: |
| 86 | |
| 87 | ```typescript |
| 88 | const sink = yield* Cloudflare.Pipelines.StreamSink(PageViews); |
| 89 | yield* Stream.fromIterable(events).pipe(Stream.run(sink)); |
| 90 | ``` |
| 91 | |
| 92 | Provide `Cloudflare.Pipelines.StreamSinkBinding` alongside `WriteStreamBinding`. |
| 93 | |
| 94 | ## Send events from outside a Worker |
| 95 | |
| 96 | The producer code stays the same on every host. Only the layer changes: |
| 97 | |
| 98 | - `WriteStreamBinding` and `StreamSinkBinding` run in Cloudflare Workers and |
| 99 | send through the native `pipelines` binding. |
| 100 | - `WriteStreamHttp` and `StreamSinkHttp` run on any host. They send to the |
| 101 | stream's HTTP endpoint with a scoped "Pipelines Send" token that Alchemy mints |
| 102 | at deploy. |
| 103 | - `WriteStreamLocal` and `StreamSinkLocal` run in scripts and Actions. They |
| 104 | send to the stream's HTTP endpoint with your current Cloudflare credentials. |
| 105 | |
| 106 | The HTTP and local layers need the stream's HTTP endpoint turned on. |
| 107 | |
| 108 | ## Turn on the HTTP endpoint |
| 109 | |
| 110 | A stream gets an HTTP ingest endpoint only when you ask for one: |
| 111 | |
| 112 | ```typescript |
| 113 | Cloudflare.Pipelines.Stream("PageViews", { http: true }); // authenticated |
| 114 | Cloudflare.Pipelines.Stream("PageViews", { |
| 115 | http: { enabled: true, authentication: false }, // public, e.g. for browsers |
| 116 | }); |
| 117 | ``` |
| 118 | |
| 119 | :::caution |
| 120 | Earlier versions of Alchemy left a stream declared without `http` with |
| 121 | Cloudflare's default, a public endpoint that anyone can write to. The next |
| 122 | deploy now closes that endpoint in place, without replacing the stream. If |
| 123 | external producers post to it, keep it public with |
| 124 | `http: { enabled: true, authentication: false }`. |
| 125 | ::: |
| 126 | |
| 127 | ## Write events to an Iceberg table |
| 128 | |
| 129 | A catalog sink writes the pipeline's output to an |
| 130 | [Iceberg table](/cloudflare/data/iceberg-tables). It needs a bucket with the |
| 131 | catalog enabled. Declare them in your stack's Effect, and pass the catalog |
| 132 | resource to the sink so it is created after the catalog is enabled: |
| 133 | |
| 134 | ```typescript |
| 135 | // alchemy.run.ts |
| 136 | import * as Alchemy from "alchemy"; |
| 137 | import * as Cloudflare from "@alchemy.run/cloudflare"; |
| 138 | import * as Config from "effect/Config"; |
| 139 | import * as Effect from "effect/Effect"; |
| 140 | import { PageViews } from "./src/Events.ts"; |
| 141 | |
| 142 | export default Alchemy.Stack( |
| 143 | "Analytics", |
| 144 | { |
| 145 | providers: Cloudflare.providers(), |
| 146 | state: Cloudflare.state(), |
| 147 | }, |
| 148 | Effect.gen(function* () { |
| 149 | const lake = yield* Cloudflare.R2.Bucket("Lake"); |
| 150 | const catalog = yield* Cloudflare.R2.DataCatalog("Catalog", { |
| 151 | bucket: lake, |
| 152 | }); |
| 153 | const views = yield* PageViews; |
| 154 | |
| 155 | const table = yield* Cloudflare.Pipelines.Sink("PageViewTable", { |
| 156 | type: "basin_catalog", |
| 157 | catalog, |
| 158 | table: { namespace: "web", name: "page_views" }, |
| 159 | token: yield* Config.Redacted("CATALOG_TOKEN"), |
| 160 | }); |
| 161 | }), |
| 162 | ); |
| 163 | ``` |
| 164 | |
| 165 | The sink creates the namespace and table on its first write. The `token` is a |
| 166 | Cloudflare API token with access to R2 and the catalog. Rotating it updates the |
| 167 | sink without replacing it. |
| 168 | |
| 169 | :::caution |
| 170 | A catalog sink can't be created for a table that already exists. Changing the |
| 171 | sink's other settings replaces the sink, and the replacement fails once the |
| 172 | table exists. Choose the table and format before the first deploy. |
| 173 | ::: |
| 174 | |
| 175 | ## Connect the stream and sink with SQL |
| 176 | |
| 177 | Add the pipeline in the same Effect: |
| 178 | |
| 179 | ```diff lang="typescript" |
| 180 | // alchemy.run.ts |
| 181 | +import * as Output from "alchemy/Output"; |
| 182 | |
| 183 | const table = yield* Cloudflare.Pipelines.Sink("PageViewTable", { /* ... */ }); |
| 184 | + |
| 185 | + yield* Cloudflare.Pipelines.Pipeline("PageViewEtl", { |
| 186 | + sql: Output.interpolate` |
| 187 | + INSERT INTO ${table.name} |
| 188 | + SELECT userId, path, at FROM ${views.name} |
| 189 | + WHERE path NOT LIKE '/health%'`, |
| 190 | + }); |
| 191 | ``` |
| 192 | |
| 193 | Alchemy validates the SQL with Cloudflare during plan, so a bad statement fails |
| 194 | before anything is replaced. Reformatting the SQL is not a change. |
| 195 | |
| 196 | ## Write files instead of a table |
| 197 | |
| 198 | An `r2` sink writes events to a bucket as JSON or Parquet files, without a |
| 199 | catalog: |
| 200 | |
| 201 | ```typescript |
| 202 | const archive = yield* Cloudflare.Pipelines.Sink("EventArchive", { |
| 203 | type: "r2", |
| 204 | config: { |
| 205 | bucket: lake.bucketName, |
| 206 | credentials: { |
| 207 | accessKeyId: yield* Config.Redacted("R2_ACCESS_KEY_ID"), |
| 208 | secretAccessKey: yield* Config.Redacted("R2_SECRET_ACCESS_KEY"), |
| 209 | }, |
| 210 | path: "events", |
| 211 | rollingPolicy: { intervalSeconds: 60 }, |
| 212 | }, |
| 213 | format: { type: "parquet", compression: "zstd" }, |
| 214 | }); |
| 215 | ``` |
| 216 | |
| 217 | `rollingPolicy` decides when the sink closes a file and starts the next one: |
| 218 | after `intervalSeconds`, after `inactivitySeconds` without new events, or when |
| 219 | the file reaches `fileSizeBytes`. Point a pipeline at it the same way, with `${archive.name}` in |
| 220 | the SQL. Unlike a catalog sink, an `r2` sink can be replaced freely. |
| 221 | |
| 222 | ## React to new files |
| 223 | |
| 224 | Pipelines has no event source of its own, because its consumer is the sink. To |
| 225 | process the files a sink writes, listen for object events on the bucket with |
| 226 | `Cloudflare.R2.BucketEventNotification`. |
| 227 | |
| 228 | ## Where next |
| 229 | |
| 230 | Related: |
| 231 | |
| 232 | - [Iceberg tables](/cloudflare/data/iceberg-tables): manage the tables a |
| 233 | catalog sink writes, and query them. |
| 234 | - [R2](/cloudflare/data/r2): the bucket a sink writes to. |
| 235 | - [K2 streams](/cloudflare/messaging/k2): a durable log your own code consumes. |
| 236 | |
| 237 | Reference: |
| 238 | |
| 239 | - [Pipelines API reference](/providers/cloudflare/pipelines): `Stream`, `Sink`, |
| 240 | `Pipeline`, `WriteStream`, and `StreamSink`. |