The best way to control Claude Code, Codex and more from any computer or phone
Mirrored from https://github.com/Camo67/t3code.git
Synced Nov 21, 06:00 UTC
| 1 | --- |
| 2 | title: Projection |
| 3 | description: How a projection consumes events into a store you own, in order per key, batched and replayable, and why it never sends commands. |
| 4 | --- |
| 5 | |
| 6 | import DAG from "../../../components/DAG.astro"; |
| 7 | |
| 8 | A projection consumes events into a store you own: a database table, a search index, a warehouse, or a [`Domain.Stream`](/domain/stream). Where a [view](/domain/view) is a read model the platform stores and serves for you, a projection writes wherever you choose and leaves serving it to you. |
| 9 | |
| 10 | It shares the view's routing and ordering, and differs from a [policy](/domain/policy) in one way that shapes everything else: a projection never sends commands. Its only effect is on its own store, so it can always be replayed. |
| 11 | |
| 12 | ## When a view isn't the right store |
| 13 | |
| 14 | A view answers one question about one key, and the model serves it. That covers most screens. Some questions don't fit that shape. An accountant wants every money movement across every account this quarter, filtered by amount and sorted by date. A search box wants to find a customer by any part of their name. A data team wants the whole history in their warehouse. Each needs a store built for its kind of query, a SQL table, a search index, a warehouse, and each is a store you run. |
| 15 | |
| 16 | A projection is how events reach a store like that. It's still a read model in the CQRS sense: built from events, and never the source of truth. Add one to a model that already has history, and it receives that history first, so its store starts complete (see [Consumers added later receive the history](/domain/model#consumers-added-later-receive-the-history)). |
| 17 | |
| 18 | <DAG |
| 19 | nodes={[ |
| 20 | { id: "events", label: "Account events", type: "the record", kind: "event", layer: 0 }, |
| 21 | { id: "view", label: "AccountSummary", type: "served for you", kind: "view", layer: 1 }, |
| 22 | { id: "ledger", label: "Ledger", type: "into your store", kind: "projection", layer: 1 }, |
| 23 | { id: "table", label: "ledger table", type: "query it your way", layer: 2 }, |
| 24 | ]} |
| 25 | edges={[ |
| 26 | { from: "events", to: "view", dashed: true }, |
| 27 | { from: "events", to: "ledger", dashed: true }, |
| 28 | { from: "ledger", to: "table" }, |
| 29 | ]} |
| 30 | caption="The same events feed a view the model serves and a table you query yourself." |
| 31 | /> |
| 32 | |
| 33 | Reach for a view first. Reach for a projection when the query needs a store a view doesn't give you, or when the data has to leave the model altogether. |
| 34 | |
| 35 | ## Declare what it consumes |
| 36 | |
| 37 | A projection names its sources, the key that partitions it, and the events it consumes. In the example bank, a ledger records every money movement on an account. |
| 38 | |
| 39 | ```typescript title="packages/domain/src/Ledger/Ledger.projection.ts" |
| 40 | export class Ledger extends Domain.Projection("Ledger", { |
| 41 | from: [Account], |
| 42 | key: Account, |
| 43 | on: [ |
| 44 | MoneyDeposited, |
| 45 | MoneyWithdrawn, |
| 46 | TransferDebited, |
| 47 | TransferCredited, |
| 48 | TransferRefunded, |
| 49 | SettlementReceived, |
| 50 | ], |
| 51 | }) {} |
| 52 | ``` |
| 53 | |
| 54 | Add it to the model's `projections`, like a policy. |
| 55 | |
| 56 | <DAG |
| 57 | nodes={[ |
| 58 | { id: "account", label: "Account a-1", type: "MoneyWithdrawn", kind: "aggregate" }, |
| 59 | { id: "ledger", label: "Ledger a-1", type: "in order", kind: "projection" }, |
| 60 | { id: "table", label: "ledger table", type: "D1" }, |
| 61 | ]} |
| 62 | edges={[ |
| 63 | { from: "account", to: "ledger", dashed: true }, |
| 64 | { from: "ledger", to: "table" }, |
| 65 | ]} |
| 66 | caption="The account commits first. Shortly after, the projection writes the movement to a table outside the model." |
| 67 | /> |
| 68 | |
| 69 | ## Implement it as a Layer |
| 70 | |
| 71 | The handler lives in a Layer, built once with everything it needs. It always receives a batch: an array of inputs for one key, in order. |
| 72 | |
| 73 | ```typescript title="packages/domain/src/Ledger/Ledger.projection.ts" |
| 74 | export const LedgerLive = Ledger.toLayer( |
| 75 | Effect.gen(function* () { |
| 76 | const store = yield* LedgerStore; |
| 77 | return Effect.fn(function* (batch) { |
| 78 | yield* store.record( |
| 79 | batch.map(({ key, event, envelope }) => ({ |
| 80 | id: `${envelope.stream}:${envelope.seq}`, |
| 81 | accountId: key.id, |
| 82 | kind: event._tag, |
| 83 | amount: event.amount, |
| 84 | balanceAfter: event.balanceAfter, |
| 85 | at: DateTime.toEpochMillis(envelope.at), |
| 86 | })), |
| 87 | ); |
| 88 | }); |
| 89 | }), |
| 90 | { batch: { size: 100, wait: "1 second" } }, |
| 91 | ); |
| 92 | ``` |
| 93 | |
| 94 | Each input carries the `key` it was routed to, the `event`, its `envelope`, the `source` it came from, and, for a view source, the view's `state` after the event. |
| 95 | |
| 96 | ## Depend on a store, not a database |
| 97 | |
| 98 | The projection asks for `LedgerStore`, a service you define in the domain package, rather than a D1 binding. The server package provides it with D1; a story provides it with memory. |
| 99 | |
| 100 | ```typescript title="packages/domain/src/Ledger/LedgerStore.ts" |
| 101 | export class LedgerStore extends Context.Service< |
| 102 | LedgerStore, |
| 103 | { |
| 104 | readonly record: (rows: ReadonlyArray<LedgerRow>) => Effect.Effect<void, never, RuntimeContext>; |
| 105 | readonly forAccount: ( |
| 106 | accountId: string, |
| 107 | ) => Effect.Effect<ReadonlyArray<LedgerRow>, never, RuntimeContext>; |
| 108 | } |
| 109 | >()("Bank/LedgerStore") {} |
| 110 | ``` |
| 111 | |
| 112 | Its methods require `RuntimeContext`, because a database binding only exists inside a running Worker. A projection's handler may require it; the platform runs handlers inside the runtime. |
| 113 | |
| 114 | ## Make every write idempotent |
| 115 | |
| 116 | Inputs arrive at least once. If a handler fails halfway, or its host restarts, the same inputs run again, so a write must be safe to repeat. The ledger keys each row by the event it came from, `stream:seq`, and ignores a row it already has. |
| 117 | |
| 118 | ```typescript title="packages/server/src/Ledger/D1LedgerStore.ts" |
| 119 | "INSERT INTO ledger (id, account_id, kind, amount, balance_after, at) VALUES (?, ?, ?, ?, ?, ?) ON CONFLICT (id) DO NOTHING", |
| 120 | ``` |
| 121 | |
| 122 | The platform helps on its side too: every key keeps an input log, and a delivery it has already accepted is dropped on arrival. |
| 123 | |
| 124 | ## Partition it by key |
| 125 | |
| 126 | `key` decides which inputs are handled in order. Inputs for one key are handled one batch at a time, in the order they were accepted; different keys run in parallel. |
| 127 | |
| 128 | <DAG |
| 129 | nodes={[ |
| 130 | { id: "a1", label: "Account a-1", type: "events", kind: "aggregate", layer: 0 }, |
| 131 | { id: "a2", label: "Account a-2", type: "events", kind: "aggregate", layer: 0 }, |
| 132 | { id: "p1", label: "Ledger a-1", type: "one at a time", kind: "projection", layer: 1 }, |
| 133 | { id: "p2", label: "Ledger a-2", type: "one at a time", kind: "projection", layer: 1 }, |
| 134 | ]} |
| 135 | edges={[ |
| 136 | { from: "a1", to: "p1", dashed: true }, |
| 137 | { from: "a2", to: "p2", dashed: true }, |
| 138 | ]} |
| 139 | caption="One ordered partition per key. Keys don't wait for each other." |
| 140 | /> |
| 141 | |
| 142 | A source keyed by the same aggregate routes itself. For any other source, `keyOf` names the key, by event tag or source name, and can send one event to several keys with named routes, exactly as a [view's `keyOf`](/domain/view#route-one-event-to-several-keys) does. Each input then carries its `route`. |
| 143 | |
| 144 | The key can also be a string Schema, such as an org id, when there is no aggregate to partition by. `keyOf` then returns the value, and the handler gets it as `key`. |
| 145 | |
| 146 | ```typescript |
| 147 | export class OrgLedger extends Domain.Projection("OrgLedger", { |
| 148 | from: [Account], |
| 149 | key: OrgId, |
| 150 | keyOf: { Account: ({ event }) => event.orgId }, |
| 151 | on: [MoneyDeposited], |
| 152 | }) {} |
| 153 | ``` |
| 154 | |
| 155 | Within a key, events from one source stay in that source's order. Events from different sources have no order between them: the model is distributed on purpose, and there is no total order to give. |
| 156 | |
| 157 | ## Batch it on its Layer |
| 158 | |
| 159 | Batching changes how a projection runs, not what it means, so it lives on the Layer. `batch: { size, wait }` hands the handler up to `size` inputs of one key at once, waiting up to `wait` for a batch to fill. Without it, each input is handled on its own, as soon as it arrives. |
| 160 | |
| 161 | ```typescript title="packages/domain/src/Ledger/Ledger.projection.ts" |
| 162 | { batch: { size: 100, wait: "1 second" } }, |
| 163 | ``` |
| 164 | |
| 165 | The handler's code is the same either way: it always receives an array. Changing the batch size is pure tuning. |
| 166 | |
| 167 | `version` is tuning too. Bump it and the projection runs again over the whole history on the next deploy, for example after its table was lost, or once the handler writes a new shape. Empty or migrate the table first if a replayed write wouldn't overwrite the old row. |
| 168 | |
| 169 | ```typescript title="packages/domain/src/Ledger/Ledger.projection.ts" |
| 170 | { batch: { size: 100, wait: "1 second" }, version: 1 }, |
| 171 | ``` |
| 172 | |
| 173 | ## It never sends commands |
| 174 | |
| 175 | A projection's effects stay in its own store. That's what makes it safe to replay: run it again over the same inputs, and the only thing that happens is the same rows are written again. |
| 176 | |
| 177 | A policy is the opposite. It acts on the world, through commands and external systems, so it handles each new event once and never replays. When you need both, split the work: the projection writes the store, and a policy on the same events sends the commands. |
| 178 | |
| 179 | <DAG |
| 180 | nodes={[ |
| 181 | { id: "event", label: "MoneyWithdrawn", type: "Account a-1", kind: "event" }, |
| 182 | { id: "ledger", label: "Ledger", type: "replayable", kind: "projection" }, |
| 183 | { id: "table", label: "ledger table", type: "your store" }, |
| 184 | { id: "review", label: "FraudReview", type: "never replays", kind: "policy" }, |
| 185 | { id: "freeze", label: "FreezeAccount", type: "acts on the world", kind: "command" }, |
| 186 | ]} |
| 187 | edges={[ |
| 188 | { from: "event", to: "ledger", dashed: true }, |
| 189 | { from: "ledger", to: "table" }, |
| 190 | { from: "event", to: "review", dashed: true }, |
| 191 | { from: "review", to: "freeze" }, |
| 192 | ]} |
| 193 | caption="The same event, consumed twice: once into a store that can be rebuilt, once into an action that can't be undone." |
| 194 | /> |
| 195 | |
| 196 | ## Read your own write |
| 197 | |
| 198 | Your store is eventually consistent with the model. A caller who has just sent a command and wants to read the result from your store waits for the projection first. |
| 199 | |
| 200 | ```typescript title="packages/domain/test/ledgerAfterWithdrawal.ts" |
| 201 | export const ledgerAfterWithdrawal = Effect.fn(function* (accountId: AccountId) { |
| 202 | const accounts = yield* Account; |
| 203 | const receipt = yield* accounts.send(accountId, new WithdrawMoney({ amount: 2_000 })); |
| 204 | yield* Ledger.caughtUp(receipt); |
| 205 | return yield* (yield* LedgerStore).forAccount(accountId); |
| 206 | }); |
| 207 | ``` |
| 208 | |
| 209 | `caughtUp` waits until the projection has handled every event in the receipt that it consumes directly from that aggregate. |
| 210 | |
| 211 | ## Serve it yourself |
| 212 | |
| 213 | A projection's store is yours, so its API is too. The example bank serves the ledger from a plain route on its Worker, authorizing the caller itself. |
| 214 | |
| 215 | ```typescript title="packages/server/src/BankWorker.ts" |
| 216 | const summary = yield* summaries.query(Account.ref(accountId)).pipe(Effect.orDie); |
| 217 | const owner = Option.getOrNull(summary)?.customerId; |
| 218 | if (actor.value._tag === "Customer" && owner !== actor.value.customerId) { |
| 219 | return HttpServerResponse.empty({ status: 404 }); |
| 220 | } |
| 221 | return yield* HttpServerResponse.json(yield* ledger.forAccount(accountId)); |
| 222 | ``` |
| 223 | |
| 224 | Clients that want to know when to refetch can follow the same events the projection consumed, through the Api. |
| 225 | |
| 226 | To have the model serve a projection's output for you, with live updates and `validate`, write a [stream](/domain/stream) instead. |
| 227 | |
| 228 | ## Test it |
| 229 | |
| 230 | Projections run for real in a [story](/domain/story). Give the story a store you can read back. |
| 231 | |
| 232 | ```typescript title="packages/domain/test/Bank.story.test.ts" |
| 233 | const ledger = memoryLedger(); |
| 234 | yield* ledgerStory( |
| 235 | given(a1, opened(), new MoneyDeposited({ amount: 10_000, balanceAfter: 10_000 })), |
| 236 | as(sam), |
| 237 | when(a1, new WithdrawMoney({ amount: 2_500 })), |
| 238 | then(a1, new MoneyWithdrawn({ amount: 2_500, balanceAfter: 7_500 })), |
| 239 | ).pipe(Effect.provide(ledger.layer)); |
| 240 | ``` |
| 241 | |
| 242 | Next, [Stream](/domain/stream): a store the model keeps and serves for a projection. |