fix(metrics): disable otel exemplars to prevent rune overflow (#11211)
* fix: disable otel exemplars to prevent prometheus rune overflow the OTel SDK View from #11208 drops server.address from http.server.* metric labels, but the OTel spec requires filtered attributes to be carried as exemplar FilteredAttributes. on subdomain gateways the server.address value (e.g. "CID.ipfs.dweb.link") combined with trace_id and span_id exceeds the 128-rune prometheus exemplar limit. - cmd/ipfs/kubo/daemon.go: add exemplar.AlwaysOffFilter to MeterProvider - docs/changelogs/v0.40.md: document exemplar disable in metrics section
Marcin Rataj committed
Feb 25, 2026 at 17:43 UTC
221741ee20581a0968b3989adab4e44fbcf9a527
2 files changed
+13
-2
cmd/ipfs/kubo/daemon.go
+9
@@ -47,6 +47,7 @@ import (
47
"go.opentelemetry.io/otel/attribute"
48
promexporter "go.opentelemetry.io/otel/exporters/prometheus"
49
sdkmetric "go.opentelemetry.io/otel/sdk/metric"
50
+ "go.opentelemetry.io/otel/sdk/metric/exemplar"
51
)
52
53
const (
@@ -239,6 +240,14 @@ func daemonFunc(req *cmds.Request, re cmds.ResponseEmitter, env cmds.Environment
240
),
241
},
242
)),
243
+ // Disable exemplars. The OTel spec requires exemplars to carry
244
+ // attributes filtered out by Views (as FilteredAttributes).
245
+ // The server.address value on subdomain gateways (e.g.
246
+ // "CID.ipfs.dweb.link") combined with trace_id and span_id
247
+ // exceeds the 128-rune Prometheus exemplar limit.
248
+ // Re-enabling exemplars requires removing all metrics that
249
+ // track server.address (the above View is not enough).
250
+ sdkmetric.WithExemplarFilter(exemplar.AlwaysOffFilter),
251
sdkmetric.WithReader(exporter),
252
)
253
otel.SetMeterProvider(meterProvider)
docs/changelogs/v0.40.md
+4
-2
@@ -307,8 +307,10 @@ Most Kubo users are unaffected by this change. It matters if you run Kubo as a p
307
308
**What changed:**
309
310
-- The unbounded `server_address` label is now dropped from all `http_server_*` metrics via an OTel SDK View.
311
-- All handlers add a `server_domain` label instead. Gateway handlers group by matching `Gateway.PublicGateways` suffix (e.g., `dweb.link`, `ipfs.io`), with `localhost`, `loopback`, or `other` for unmatched hosts. The RPC API and Libp2p Gateway handlers use fixed values (`api`, `libp2p`).
310
+- `http_server_*` metrics replace the unbounded `server_address` label with a new `server_domain` label that groups requests by gateway domain:
311
+ - Gateway: matched [`Gateway.PublicGateways`](https://github.com/ipfs/kubo/blob/master/docs/config.md#gatewaypublicgateways) suffix (e.g., `dweb.link`, `ipfs.io`), or `localhost`, `loopback`, `other`
312
+ - RPC API: `api` / Libp2p Gateway: `libp2p`
313
+- Prometheus exemplars are disabled to prevent log noise from long subdomain hostnames. Tracing spans are unaffected.
314
315
If you use [Rainbow](https://github.com/ipfs/rainbow) for your public gateway (recommended), this issue never applied to you -- Rainbow uses its own low-cardinality HTTP metrics.
316