@devries-cameron20 / t3code Mirror · read-only
Fork 0

The best way to control Claude Code, Codex and more from any computer or phone

Mirrored from https://github.com/Camo67/t3code.git

Synced Nov 21, 06:00 UTC

main
mdx 194 lines 5.91 KB
Raw
1 ---
2 title: Iceberg tables
3 description: Turn an R2 bucket into an Apache Iceberg warehouse with Cloudflare's catalog, manage tables with schema evolution and maintenance, and query them from Spark, DuckDB, or PyIceberg.
4 ---
5
6 Cloudflare's **catalog** (formerly R2 Data Catalog) turns an R2 bucket into an
7 Apache Iceberg warehouse. Iceberg stores a table as data files plus metadata,
8 so any engine that speaks the Iceberg REST protocol can read and write it:
9 Spark, DuckDB, PyIceberg, Snowflake, Trino, and others.
10
11 The catalog is part of **Basin**, Cloudflare's analytics stack on R2. Tables
12 get into it in two ways:
13
14 - A [Pipelines](/cloudflare/data/pipelines) catalog sink creates a table and
15 writes events into it.
16 - `Cloudflare.Basin.Table` creates and manages a table directly, for tables
17 your own code or engines write.
18
19 This page covers enabling the catalog, managing tables, and connecting query
20 engines.
21
22 ## Enable the catalog on a bucket
23
24 ```typescript
25 // alchemy.run.ts
26 const lake = yield* Cloudflare.R2.Bucket("Lake");
27
28 const catalog = yield* Cloudflare.Basin.Catalog("Catalog", {
29 bucket: lake,
30 });
31 ```
32
33 `Basin.Catalog` is the same resource as `Cloudflare.R2.DataCatalog`. It
34 outputs:
35
36 - `catalogUri`: the Iceberg REST endpoint, for example
37 `https://catalog.cloudflarestorage.com/<account>/<bucket>`.
38 - `name`: the warehouse name, `<account>_<bucket>`.
39
40 ## Turn on catalog maintenance
41
42 Iceberg tables accumulate small files and old snapshots. The catalog can
43 compact files and expire snapshots for every table in the warehouse:
44
45 ```typescript
46 const catalog = yield* Cloudflare.Basin.Catalog("Catalog", {
47 bucket: lake,
48 compaction: { state: "enabled", targetSizeMb: "128" },
49 snapshotExpiration: { state: "enabled", maxSnapshotAge: "7d", minSnapshotsToKeep: 10 },
50 token: yield* Config.Redacted("CATALOG_MAINTENANCE_TOKEN"),
51 });
52 ```
53
54 The maintenance jobs run with `token`, a Cloudflare API token with access to R2
55 and the catalog. They stay pending until a token is set. Only the fields you
56 set are enforced; the rest keep Cloudflare's defaults.
57
58 ## Create a table
59
60 `Cloudflare.Basin.Table` creates a namespace and table through the catalog's
61 Iceberg REST API, with columns from an Effect Schema:
62
63 ```typescript
64 import * as Schema from "effect/Schema";
65
66 const Order = Schema.Struct({
67 orderId: Schema.String,
68 amountCents: Schema.Number,
69 at: Schema.Date,
70 note: Schema.optional(Schema.String),
71 });
72
73 const orders = yield* Cloudflare.Basin.Table("Orders", {
74 catalog,
75 namespace: "sales",
76 schema: Order,
77 partitionBy: ["day(at)"],
78 properties: { "write.format.default": "parquet" },
79 });
80 ```
81
82 The table name is generated from the stack, stage, and logical ID unless you
83 set `name`. The table outputs `identifier` (for example `sales.orders`),
84 `tableUuid`, `location`, and `metadataLocation`.
85
86 `partitionBy` takes Iceberg transforms: a column name, `day(at)`, `month(at)`,
87 `bucket[16](orderId)`, and so on. `properties` are Iceberg table properties,
88 updated in place.
89
90 The table authenticates with your deploying Cloudflare credentials. Pass `token`
91 to use a different one.
92
93 ## Evolve the schema
94
95 Adding an optional column evolves the table in place. Existing data stays, and
96 the new column reads as empty for old rows:
97
98 ```diff lang="typescript"
99 const Order = Schema.Struct({
100 orderId: Schema.String,
101 amountCents: Schema.Number,
102 at: Schema.Date,
103 note: Schema.optional(Schema.String),
104 + coupon: Schema.optional(Schema.String),
105 });
106 ```
107
108 A change that would lose data fails the plan before anything is applied:
109 dropping or narrowing a column, or adding a required one. Changing `namespace`,
110 `name`, or `partitionBy` also fails the plan, because each would mean dropping
111 the table.
112
113 ## Tune maintenance per table
114
115 `maintenance` overrides the catalog's settings for one table:
116
117 ```typescript
118 const orders = yield* Cloudflare.Basin.Table("Orders", {
119 catalog,
120 namespace: "sales",
121 schema: Order,
122 maintenance: {
123 compaction: { targetSizeMb: "256" },
124 snapshotExpiration: { olderThan: "3 days", retainLast: 5 },
125 },
126 });
127 ```
128
129 Per-table snapshot expiration only works after snapshot expiration is enabled
130 on the catalog.
131
132 ## Choose what happens on destroy
133
134 Destroying a `Table` resource leaves the table and its data in place by
135 default, so removing it from your program never deletes data:
136
137 ```typescript
138 Cloudflare.Basin.Table("Orders", { catalog, namespace: "sales", schema: Order, delete: "retain" }); // default
139 Cloudflare.Basin.Table("Orders", { catalog, namespace: "sales", schema: Order, delete: "drop" }); // drop the table, keep its files
140 Cloudflare.Basin.Table("Orders", { catalog, namespace: "sales", schema: Order, delete: "purge" }); // drop the table and its files
141 ```
142
143 ## Query from other engines
144
145 Any Iceberg REST client connects with the catalog's URI, the warehouse name,
146 and a Cloudflare API token with access to R2 and the catalog. Export the
147 first two from your stack:
148
149 ```typescript
150 return {
151 catalogUri: catalog.catalogUri,
152 warehouse: catalog.name,
153 };
154 ```
155
156 Then point the client at them. With PyIceberg:
157
158 ```python
159 from pyiceberg.catalog.rest import RestCatalog
160
161 catalog = RestCatalog(
162 name="lake",
163 uri=CATALOG_URI,
164 warehouse=WAREHOUSE,
165 token=CLOUDFLARE_API_TOKEN,
166 )
167 orders = catalog.load_table(("sales", "orders"))
168 print(orders.scan().to_pandas())
169 ```
170
171 From Effect code, `@distilled.cloud/iceberg` is a typed client for the same
172 API:
173
174 ```typescript
175 import * as Iceberg from "@distilled.cloud/iceberg";
176
177 const table = yield* Iceberg.loadTable({ namespace: "sales", table: "orders" }).pipe(
178 Effect.provide(Iceberg.IcebergProtocol),
179 Effect.provide(Iceberg.fromCatalogConfig({ uri: catalogUri, warehouse, token })),
180 );
181 ```
182
183 ## Where next
184
185 Related:
186
187 - [Pipelines](/cloudflare/data/pipelines): ingest events into a table with a
188 catalog sink.
189 - [R2](/cloudflare/data/r2): the bucket under the warehouse.
190
191 Reference:
192
193 - [Basin API reference](/providers/cloudflare/basin): `Catalog` and `Table`.
194 - [R2 API reference](/providers/cloudflare/r2): `DataCatalog`.