The best way to control Claude Code, Codex and more from any computer or phone
| 1 | --- |
| 2 | title: Iceberg tables |
| 3 | description: Turn an R2 bucket into an Apache Iceberg warehouse with Cloudflare's catalog, manage tables with schema evolution and maintenance, and query them from Spark, DuckDB, or PyIceberg. |
| 4 | --- |
| 5 | |
| 6 | Cloudflare's **catalog** (formerly R2 Data Catalog) turns an R2 bucket into an |
| 7 | Apache Iceberg warehouse. Iceberg stores a table as data files plus metadata, |
| 8 | so any engine that speaks the Iceberg REST protocol can read and write it: |
| 9 | Spark, DuckDB, PyIceberg, Snowflake, Trino, and others. |
| 10 | |
| 11 | The catalog is part of **Basin**, Cloudflare's analytics stack on R2. Tables |
| 12 | get into it in two ways: |
| 13 | |
| 14 | - A [Pipelines](/cloudflare/data/pipelines) catalog sink creates a table and |
| 15 | writes events into it. |
| 16 | - `Cloudflare.Basin.Table` creates and manages a table directly, for tables |
| 17 | your own code or engines write. |
| 18 | |
| 19 | This page covers enabling the catalog, managing tables, and connecting query |
| 20 | engines. |
| 21 | |
| 22 | ## Enable the catalog on a bucket |
| 23 | |
| 24 | ```typescript |
| 25 | // alchemy.run.ts |
| 26 | const lake = yield* Cloudflare.R2.Bucket("Lake"); |
| 27 | |
| 28 | const catalog = yield* Cloudflare.Basin.Catalog("Catalog", { |
| 29 | bucket: lake, |
| 30 | }); |
| 31 | ``` |
| 32 | |
| 33 | `Basin.Catalog` is the same resource as `Cloudflare.R2.DataCatalog`. It |
| 34 | outputs: |
| 35 | |
| 36 | - `catalogUri`: the Iceberg REST endpoint, for example |
| 37 | `https://catalog.cloudflarestorage.com/<account>/<bucket>`. |
| 38 | - `name`: the warehouse name, `<account>_<bucket>`. |
| 39 | |
| 40 | ## Turn on catalog maintenance |
| 41 | |
| 42 | Iceberg tables accumulate small files and old snapshots. The catalog can |
| 43 | compact files and expire snapshots for every table in the warehouse: |
| 44 | |
| 45 | ```typescript |
| 46 | const catalog = yield* Cloudflare.Basin.Catalog("Catalog", { |
| 47 | bucket: lake, |
| 48 | compaction: { state: "enabled", targetSizeMb: "128" }, |
| 49 | snapshotExpiration: { state: "enabled", maxSnapshotAge: "7d", minSnapshotsToKeep: 10 }, |
| 50 | token: yield* Config.Redacted("CATALOG_MAINTENANCE_TOKEN"), |
| 51 | }); |
| 52 | ``` |
| 53 | |
| 54 | The maintenance jobs run with `token`, a Cloudflare API token with access to R2 |
| 55 | and the catalog. They stay pending until a token is set. Only the fields you |
| 56 | set are enforced; the rest keep Cloudflare's defaults. |
| 57 | |
| 58 | ## Create a table |
| 59 | |
| 60 | `Cloudflare.Basin.Table` creates a namespace and table through the catalog's |
| 61 | Iceberg REST API, with columns from an Effect Schema: |
| 62 | |
| 63 | ```typescript |
| 64 | import * as Schema from "effect/Schema"; |
| 65 | |
| 66 | const Order = Schema.Struct({ |
| 67 | orderId: Schema.String, |
| 68 | amountCents: Schema.Number, |
| 69 | at: Schema.Date, |
| 70 | note: Schema.optional(Schema.String), |
| 71 | }); |
| 72 | |
| 73 | const orders = yield* Cloudflare.Basin.Table("Orders", { |
| 74 | catalog, |
| 75 | namespace: "sales", |
| 76 | schema: Order, |
| 77 | partitionBy: ["day(at)"], |
| 78 | properties: { "write.format.default": "parquet" }, |
| 79 | }); |
| 80 | ``` |
| 81 | |
| 82 | The table name is generated from the stack, stage, and logical ID unless you |
| 83 | set `name`. The table outputs `identifier` (for example `sales.orders`), |
| 84 | `tableUuid`, `location`, and `metadataLocation`. |
| 85 | |
| 86 | `partitionBy` takes Iceberg transforms: a column name, `day(at)`, `month(at)`, |
| 87 | `bucket[16](orderId)`, and so on. `properties` are Iceberg table properties, |
| 88 | updated in place. |
| 89 | |
| 90 | The table authenticates with your deploying Cloudflare credentials. Pass `token` |
| 91 | to use a different one. |
| 92 | |
| 93 | ## Evolve the schema |
| 94 | |
| 95 | Adding an optional column evolves the table in place. Existing data stays, and |
| 96 | the new column reads as empty for old rows: |
| 97 | |
| 98 | ```diff lang="typescript" |
| 99 | const Order = Schema.Struct({ |
| 100 | orderId: Schema.String, |
| 101 | amountCents: Schema.Number, |
| 102 | at: Schema.Date, |
| 103 | note: Schema.optional(Schema.String), |
| 104 | + coupon: Schema.optional(Schema.String), |
| 105 | }); |
| 106 | ``` |
| 107 | |
| 108 | A change that would lose data fails the plan before anything is applied: |
| 109 | dropping or narrowing a column, or adding a required one. Changing `namespace`, |
| 110 | `name`, or `partitionBy` also fails the plan, because each would mean dropping |
| 111 | the table. |
| 112 | |
| 113 | ## Tune maintenance per table |
| 114 | |
| 115 | `maintenance` overrides the catalog's settings for one table: |
| 116 | |
| 117 | ```typescript |
| 118 | const orders = yield* Cloudflare.Basin.Table("Orders", { |
| 119 | catalog, |
| 120 | namespace: "sales", |
| 121 | schema: Order, |
| 122 | maintenance: { |
| 123 | compaction: { targetSizeMb: "256" }, |
| 124 | snapshotExpiration: { olderThan: "3 days", retainLast: 5 }, |
| 125 | }, |
| 126 | }); |
| 127 | ``` |
| 128 | |
| 129 | Per-table snapshot expiration only works after snapshot expiration is enabled |
| 130 | on the catalog. |
| 131 | |
| 132 | ## Choose what happens on destroy |
| 133 | |
| 134 | Destroying a `Table` resource leaves the table and its data in place by |
| 135 | default, so removing it from your program never deletes data: |
| 136 | |
| 137 | ```typescript |
| 138 | Cloudflare.Basin.Table("Orders", { catalog, namespace: "sales", schema: Order, delete: "retain" }); // default |
| 139 | Cloudflare.Basin.Table("Orders", { catalog, namespace: "sales", schema: Order, delete: "drop" }); // drop the table, keep its files |
| 140 | Cloudflare.Basin.Table("Orders", { catalog, namespace: "sales", schema: Order, delete: "purge" }); // drop the table and its files |
| 141 | ``` |
| 142 | |
| 143 | ## Query from other engines |
| 144 | |
| 145 | Any Iceberg REST client connects with the catalog's URI, the warehouse name, |
| 146 | and a Cloudflare API token with access to R2 and the catalog. Export the |
| 147 | first two from your stack: |
| 148 | |
| 149 | ```typescript |
| 150 | return { |
| 151 | catalogUri: catalog.catalogUri, |
| 152 | warehouse: catalog.name, |
| 153 | }; |
| 154 | ``` |
| 155 | |
| 156 | Then point the client at them. With PyIceberg: |
| 157 | |
| 158 | ```python |
| 159 | from pyiceberg.catalog.rest import RestCatalog |
| 160 | |
| 161 | catalog = RestCatalog( |
| 162 | name="lake", |
| 163 | uri=CATALOG_URI, |
| 164 | warehouse=WAREHOUSE, |
| 165 | token=CLOUDFLARE_API_TOKEN, |
| 166 | ) |
| 167 | orders = catalog.load_table(("sales", "orders")) |
| 168 | print(orders.scan().to_pandas()) |
| 169 | ``` |
| 170 | |
| 171 | From Effect code, `@distilled.cloud/iceberg` is a typed client for the same |
| 172 | API: |
| 173 | |
| 174 | ```typescript |
| 175 | import * as Iceberg from "@distilled.cloud/iceberg"; |
| 176 | |
| 177 | const table = yield* Iceberg.loadTable({ namespace: "sales", table: "orders" }).pipe( |
| 178 | Effect.provide(Iceberg.IcebergProtocol), |
| 179 | Effect.provide(Iceberg.fromCatalogConfig({ uri: catalogUri, warehouse, token })), |
| 180 | ); |
| 181 | ``` |
| 182 | |
| 183 | ## Where next |
| 184 | |
| 185 | Related: |
| 186 | |
| 187 | - [Pipelines](/cloudflare/data/pipelines): ingest events into a table with a |
| 188 | catalog sink. |
| 189 | - [R2](/cloudflare/data/r2): the bucket under the warehouse. |
| 190 | |
| 191 | Reference: |
| 192 | |
| 193 | - [Basin API reference](/providers/cloudflare/basin): `Catalog` and `Table`. |
| 194 | - [R2 API reference](/providers/cloudflare/r2): `DataCatalog`. |