@devries-cameron20 / t3code Mirror · read-only
Fork 0

The best way to control Claude Code, Codex and more from any computer or phone

Mirrored from https://github.com/Camo67/t3code.git

Synced Nov 21, 06:00 UTC

main
mdx 179 lines 6.23 KB
Raw
1 ---
2 title: Datasets & ingest
3 description: Axiom datasets per OTel signal with OTLP endpoints as outputs, least-privilege ingest tokens, and per-stage naming with Stack.useSync.
4 ---
5
6 A `Dataset` is Axiom's top-level container — it stores your events,
7 logs, traces, or metrics. An `ApiToken` is the credential your
8 runtime ships with. Both are one-line resources, declared in the
9 same Stack as the Workers and Functions that emit the data.
10
11 ## Create a dataset
12
13 ```typescript
14 import * as Axiom from "@alchemy.run/axiom";
15
16 const logs = yield* Axiom.Dataset("logs", {
17 name: "my-app-logs",
18 kind: "otel:logs:v1",
19 description: "Application logs from prod workers",
20 retentionDays: 30,
21 useRetentionPeriod: true,
22 });
23 ```
24
25 `description`, `retentionDays`, and `useRetentionPeriod` update in
26 place, so a retention change is a reviewable diff.
27
28 :::caution[Name and kind changes delete data]
29 `name` is the dataset's stable identifier in Axiom and `kind` is
30 fixed at creation — changing either triggers a **replacement**: the
31 old dataset is deleted along with everything ingested into it. Pick
32 both up front, and never rename a dataset that holds data you care
33 about.
34 :::
35
36 ## One dataset per signal
37
38 `kind` determines the schema and how the data renders in the Axiom
39 UI. For OTel pipelines, create one dataset per signal:
40
41 ```typescript
42 const traces = yield* Axiom.Dataset("traces", { name: "app-traces", kind: "otel:traces:v1" });
43 const logs = yield* Axiom.Dataset("logs", { name: "app-logs", kind: "otel:logs:v1" });
44 const metrics = yield* Axiom.Dataset("metrics", { name: "app-metrics", kind: "otel:metrics:v1" });
45 ```
46
47 The default `kind` is `axiom:events:v1` — Axiom's general-purpose
48 event store — for anything that isn't OTLP.
49
50 ## OTLP endpoints as outputs
51
52 Each dataset exposes Axiom's OTLP/HTTP endpoints as output
53 attributes, and alchemy runtimes have
54 [telemetry built in](/infrastructure-as-effects/telemetry) — wiring
55 an exporter is providing the `Axiom.Telemetry` binding layer on the
56 Function/Worker:
57
58 ```typescript
59 // src/observability.ts — declared at module scope so the Worker can import them
60 export const Traces = Axiom.Dataset("Traces", { name: "app-traces", kind: "otel:traces:v1" });
61 export const Logs = Axiom.Dataset("Logs", { name: "app-logs", kind: "otel:logs:v1" });
62 export const Ingest = Axiom.ApiToken("Ingest", {
63 name: "app-ingest",
64 datasetCapabilities: {
65 "app-traces": { ingest: ["create"] },
66 "app-logs": { ingest: ["create"] },
67 },
68 });
69
70 // src/worker.ts — on the Worker's constructor Effect, merged with its other binding layers:
71 Effect.provide(
72 Layer.mergeAll(
73 Cloudflare.R2.ReadWriteBucketBinding,
74 Axiom.Telemetry({ token: Ingest, traces: Traces, logs: Logs }),
75 ),
76 )
77 ```
78
79 Building the layer binds each dataset's endpoint and the ingest
80 token's `Authorization` header (as a secret) onto the host; at
81 runtime the built-in exporter ships each signal to its dataset,
82 flushed per request. See the
83 [end-to-end guide](/cloudflare/observability/axiom-observability)
84 for the full pattern.
85
86 - `otelTracesEndpoint` / `otelLogsEndpoint` / `otelMetricsEndpoint` —
87 the per-signal OTLP/HTTP URLs.
88 - `otelEndpoint` — the root; most exporters auto-append the signal
89 path.
90 - `otelHeaders` — the non-secret headers Axiom expects, i.e.
91 `X-Axiom-Dataset: <name>`, or `X-Axiom-Metrics-Dataset: <name>` for a
92 `otel:metrics:v1` dataset. The `Authorization: Bearer` header is
93 deliberately excluded so no secret lives in dataset state — that's
94 what the ingest token below is for.
95
96 ## Mint an ingest token
97
98 An `ApiToken` scoped with `datasetCapabilities` can ingest into
99 exactly the datasets you list and nothing else:
100
101 ```typescript
102 const ingest = yield* Axiom.ApiToken("ingest", {
103 name: "prod-ingest",
104 description: "OTEL collector ingest",
105 datasetCapabilities: {
106 "my-app-traces": { ingest: ["create"] },
107 },
108 });
109 ```
110
111 The same shape covers read-side credentials — for example a
112 query-only token for Grafana:
113
114 ```typescript
115 yield* Axiom.ApiToken("query", {
116 name: "grafana-reader",
117 datasetCapabilities: {
118 "my-app-traces": { query: ["read"] },
119 "my-app-logs": { query: ["read"] },
120 },
121 });
122 ```
123
124 ## Token rotation
125
126 Axiom has no update endpoint for tokens, so capabilities are pinned
127 at creation: changing **any** prop replaces the token and mints a
128 new bearer value; identical props never rotate it. That makes
129 rotation a deploy — tweak the `description` (or any other field)
130 and the next deploy creates a fresh token, rewires every consumer,
131 and deletes the old one.
132
133 :::caution[The bearer persists in alchemy state]
134 Axiom returns the raw token exactly once, at create time. Alchemy
135 captures it into `ingest.token` as a `Redacted<string>` and persists
136 it in resource state — otherwise later deploys couldn't rewire it
137 into consumers. Treat your state store as sensitive: anyone with
138 read access can recover the bearer. Pass `ingest.token` to secret
139 sinks (a Worker env var becomes a `secret_text` binding; or a
140 [secrets store](/cloudflare/security/secrets-env)) rather than ever unwrapping it
141 yourself.
142 :::
143
144 ## Per-stage datasets
145
146 Dataset names are global to your Axiom org, so give each stage its
147 own. `Stack.useSync` computes props from the resolved stage — handy
148 at module scope where there's no `yield*` available:
149
150 ```typescript
151 import * as Axiom from "@alchemy.run/axiom";
152 import { Stack } from "alchemy/Stack";
153
154 export const Traces = Axiom.Dataset(
155 "Traces",
156 Stack.useSync(({ stage }) => ({
157 name: `${stage}-traces`,
158 kind: "otel:traces:v1" as const,
159 description: `OTEL traces for stage '${stage}'`,
160 retentionDays: 30,
161 })),
162 );
163 ```
164
165 Now `dev-sam` telemetry never lands in the `prod` dataset, and
166 tearing down a preview stage deletes its data with it. See
167 [Stack](/infrastructure-as-code/stack) for the full `useSync` semantics.
168
169 ## Where next
170
171 - [Ship Worker telemetry to Axiom](/cloudflare/observability/axiom-observability) —
172 the end-to-end guide: datasets, token, env wiring, and the
173 zero-exporter Cloudflare push path.
174 - [Alerting](/axiom/guides/alerting) — monitors and notifiers over
175 the data you just ingested.
176 - [Telemetry](/infrastructure-as-effects/telemetry) — the built-in
177 OTel exporters and how they flush per request.
178 - Reference: [Dataset](/providers/axiom/dataset) ·
179 [ApiToken](/providers/axiom/apitoken)