| 1 | ## Kubo metrics |
| 2 | |
| 3 | By default, a Prometheus endpoint is exposed by Kubo at `http://127.0.0.1:5001/debug/metrics/prometheus`. |
| 4 | |
| 5 | It includes default [Prometheus Go client metrics](https://prometheus.io/docs/guides/go-application/) + Kubo-specific metrics listed below. |
| 6 | |
| 7 | ### Table of Contents |
| 8 | |
| 9 | - [DHT RPC](#dht-rpc) |
| 10 | - [Inbound RPC metrics](#inbound-rpc-metrics) |
| 11 | - [Outbound RPC metrics](#outbound-rpc-metrics) |
| 12 | - [Provide](#provide) |
| 13 | - [Legacy Provider](#legacy-provider) |
| 14 | - [DHT Provider](#dht-provider) |
| 15 | - [Gateway (`boxo/gateway`)](#gateway-boxogateway) |
| 16 | - [HTTP metrics](#http-metrics) |
| 17 | - [Blockstore cache metrics](#blockstore-cache-metrics) |
| 18 | - [Backend metrics](#backend-metrics) |
| 19 | - [Generic HTTP Servers](#generic-http-servers) |
| 20 | - [Core HTTP metrics](#core-http-metrics-ipfs_http_) |
| 21 | - [HTTP Server metrics](#http-server-metrics-http_server_) |
| 22 | - [OpenTelemetry Metadata](#opentelemetry-metadata) |
| 23 | |
| 24 | > [!WARNING] |
| 25 | > This documentation is incomplete. For an up-to-date list of metrics available at daemon startup, see [test/sharness/t0119-prometheus-data/prometheus_metrics_added_by_measure_profile](https://github.com/ipfs/kubo/blob/master/test/sharness/t0119-prometheus-data/prometheus_metrics_added_by_measure_profile). |
| 26 | > |
| 27 | > Additional metrics may appear during runtime as some components (like boxo/gateway) register metrics only after their first event occurs (e.g., HTTP request/response). |
| 28 | |
| 29 | ## DHT RPC |
| 30 | |
| 31 | Metrics from `go-libp2p-kad-dht` for DHT RPC operations: |
| 32 | |
| 33 | ### Inbound RPC metrics |
| 34 | |
| 35 | - `rpc_inbound_messages_total` - Counter: total messages received per RPC |
| 36 | - `rpc_inbound_message_errors_total` - Counter: total errors for received messages |
| 37 | - `rpc_inbound_bytes_[bucket|sum|count]` - Histogram: distribution of received bytes per RPC |
| 38 | - `rpc_inbound_request_latency_[bucket|sum|count]` - Histogram: latency distribution for inbound RPCs |
| 39 | |
| 40 | ### Outbound RPC metrics |
| 41 | |
| 42 | - `rpc_outbound_messages_total` - Counter: total messages sent per RPC |
| 43 | - `rpc_outbound_message_errors_total` - Counter: total errors for sent messages |
| 44 | - `rpc_outbound_requests_total` - Counter: total requests sent |
| 45 | - `rpc_outbound_request_errors_total` - Counter: total errors for sent requests |
| 46 | - `rpc_outbound_bytes_[bucket|sum|count]` - Histogram: distribution of sent bytes per RPC |
| 47 | - `rpc_outbound_request_latency_[bucket|sum|count]` - Histogram: latency distribution for outbound RPCs |
| 48 | |
| 49 | ## Provide |
| 50 | |
| 51 | ### Legacy Provider |
| 52 | |
| 53 | Metrics for the legacy provider system when `Provide.DHT.SweepEnabled=false`: |
| 54 | |
| 55 | - `provider_reprovider_provide_count` - Counter: total successful provide operations since node startup |
| 56 | - `provider_reprovider_reprovide_count` - Counter: total reprovide sweep operations since node startup |
| 57 | |
| 58 | ### DHT Provider |
| 59 | |
| 60 | Metrics for the DHT provider system when `Provide.DHT.SweepEnabled=true`: |
| 61 | |
| 62 | - `provider_provides_total` - Counter: total successful provide operations since node startup (includes both one-time provides and periodic provides done on `Provide.DHT.Interval`) |
| 63 | |
| 64 | > [!NOTE] |
| 65 | > These metrics are exposed by [go-libp2p-kad-dht](https://github.com/libp2p/go-libp2p-kad-dht/). You can enable debug logging for DHT provider activity with `GOLOG_LOG_LEVEL=dht/provider=debug`. |
| 66 | |
| 67 | ## Gateway (`boxo/gateway`) |
| 68 | |
| 69 | > [!TIP] |
| 70 | > These metrics are limited to [IPFS Gateway](https://specs.ipfs.tech/http-gateways/) endpoints. For general HTTP metrics across all endpoints, consider using a reverse proxy. |
| 71 | |
| 72 | Gateway metrics appear after the first HTTP request is processed: |
| 73 | |
| 74 | ### HTTP metrics |
| 75 | |
| 76 | - `ipfs_http_gw_responses_total{code}` - Counter: total HTTP responses by status code |
| 77 | - `ipfs_http_gw_retrieval_timeouts_total{code,truncated}` - Counter: requests that timed out during content retrieval |
| 78 | - `ipfs_http_gw_concurrent_requests` - Gauge: number of requests currently being processed |
| 79 | |
| 80 | ### Blockstore cache metrics |
| 81 | |
| 82 | - `ipfs_http_blockstore_cache_hit` - Counter: global block cache hits |
| 83 | - `ipfs_http_blockstore_cache_requests` - Counter: global block cache requests |
| 84 | |
| 85 | ### Backend metrics |
| 86 | |
| 87 | - `ipfs_gw_backend_api_call_duration_seconds_[bucket|sum|count]{backend_method}` - Histogram: time spent in IPFSBackend API calls |
| 88 | |
| 89 | ## Generic HTTP Servers |
| 90 | |
| 91 | > [!TIP] |
| 92 | > The metrics below are not very useful and exist mostly for historical reasons. If you need non-gateway HTTP metrics, it's better to put a reverse proxy in front of Kubo and use its metrics. |
| 93 | |
| 94 | ### Core HTTP metrics (`ipfs_http_*`) |
| 95 | |
| 96 | Prometheus metrics for the HTTP API exposed at port 5001: |
| 97 | |
| 98 | - `ipfs_http_requests_total{method,code,handler}` - Counter: total HTTP requests (Legacy - new metrics are provided by boxo/gateway for gateway traffic) |
| 99 | - `ipfs_http_request_duration_seconds[_sum|_count]{handler}` - Summary: request processing duration |
| 100 | - `ipfs_http_request_size_bytes[_sum|_count]{handler}` - Summary: request body sizes |
| 101 | - `ipfs_http_response_size_bytes[_sum|_count]{handler}` - Summary: response body sizes |
| 102 | |
| 103 | ### HTTP Server metrics (`http_server_*`) |
| 104 | |
| 105 | Additional HTTP instrumentation for all handlers (Gateway, API commands, etc.): |
| 106 | |
| 107 | - `http_server_request_body_size_bytes_[bucket|count|sum]` - Histogram: distribution of request body sizes |
| 108 | - `http_server_request_duration_seconds_[bucket|count|sum]` - Histogram: distribution of request processing times |
| 109 | - `http_server_response_body_size_bytes_[bucket|count|sum]` - Histogram: distribution of response body sizes |
| 110 | |
| 111 | These metrics are automatically added to Gateway handlers, Hostname Gateway, Libp2p Gateway, and API command handlers. |
| 112 | |
| 113 | > [!NOTE] |
| 114 | > The `server_address` label from `otelhttp` is dropped via an OTel SDK View to prevent cardinality explosion on subdomain gateways (where each unique `Host` header creates a new time series). All handlers include a `server_domain` label instead: |
| 115 | > |
| 116 | > - Gateway and Hostname Gateway handlers group requests by their matching [`Gateway.PublicGateways`](config.md#gatewaypublicgateways) domain suffix (e.g., `dweb.link`, `ipfs.io`). Unmatched hosts are labeled `localhost`, `loopback`, or `other`. |
| 117 | > - The RPC API handler uses `api`. |
| 118 | > - The Libp2p Gateway handler uses `libp2p`. |
| 119 | |
| 120 | ## OpenTelemetry Metadata |
| 121 | |
| 122 | Kubo uses Prometheus for metrics collection for historical reasons, but OpenTelemetry metrics are automatically exposed through the same Prometheus endpoint. These metadata metrics provide context about the instrumentation: |
| 123 | |
| 124 | - `otel_scope_name`, `otel_scope_version`, `otel_scope_schema_url` - Per-metric labels identifying the instrumentation library that produced each metric |
| 125 | - `target_info` - Service metadata including version and instance information |