| 1 | # Provide Stats |
| 2 | |
| 3 | The `ipfs provide stat` command gives you statistics about your local provide |
| 4 | system. This file provides a detailed explanation of the metrics reported by |
| 5 | this command. |
| 6 | |
| 7 | ## Understanding the Metrics |
| 8 | |
| 9 | The statistics are organized into three types of measurements: |
| 10 | |
| 11 | ### Per-worker rates |
| 12 | |
| 13 | Metrics like "CIDs reprovided/min/worker" measure the throughput of a single |
| 14 | worker processing one region. To estimate total system throughput, multiply by |
| 15 | the number of active workers of that type (see [Workers stats](#workers-stats)). |
| 16 | |
| 17 | Example: If "CIDs reprovided/min/worker" shows 100 and you have 10 active |
| 18 | periodic workers, your total reprovide throughput is approximately 1,000 |
| 19 | CIDs/min. |
| 20 | |
| 21 | ### Per-region averages |
| 22 | |
| 23 | Metrics like "Avg CIDs/reprovide" measure properties of the work units (keyspace |
| 24 | regions). These represent the average size or characteristics of a region, not a |
| 25 | rate. Do NOT multiply these by worker count. |
| 26 | |
| 27 | Example: "Avg CIDs/reprovide: 250,000" means each region contains an average of |
| 28 | 250,000 CIDs that get reprovided together as a batch. |
| 29 | |
| 30 | ### System totals |
| 31 | |
| 32 | Metrics like "Total CIDs provided" are cumulative counts since node startup. |
| 33 | These aggregate all work across all workers over time. |
| 34 | |
| 35 | ## Connectivity |
| 36 | |
| 37 | ### Status |
| 38 | |
| 39 | Current connectivity status (`online`, `disconnected`, or `offline`) and when |
| 40 | it last changed (see [provide connectivity |
| 41 | status](./config.md#providedhtofflinedelay)). |
| 42 | |
| 43 | ## Queues |
| 44 | |
| 45 | ### Provide queue |
| 46 | |
| 47 | Number of CIDs waiting for initial provide, and the number of keyspace regions |
| 48 | they're grouped into. |
| 49 | |
| 50 | ### Reprovide queue |
| 51 | |
| 52 | Number of regions with overdue reprovides. These regions missed their scheduled |
| 53 | reprovide time and will be processed as soon as possible. If decreasing, the |
| 54 | node is recovering from downtime. If increasing, either the node is offline or |
| 55 | the provide system needs more workers (see |
| 56 | [`Provide.DHT.MaxWorkers`](./config.md#providedhtmaxworkers) |
| 57 | and |
| 58 | [`Provide.DHT.DedicatedPeriodicWorkers`](./config.md#providedhtdedicatedperiodicworkers)). |
| 59 | |
| 60 | ## Schedule |
| 61 | |
| 62 | ### CIDs scheduled |
| 63 | |
| 64 | Total CIDs scheduled for reprovide. |
| 65 | |
| 66 | ### Regions scheduled |
| 67 | |
| 68 | Number of keyspace regions scheduled for reprovide. Each CID is mapped to a |
| 69 | specific region, and all CIDs within the same region are reprovided together as |
| 70 | a batch for efficient processing. |
| 71 | |
| 72 | ### Avg prefix length |
| 73 | |
| 74 | Average length of binary prefixes identifying the scheduled regions. Each |
| 75 | keyspace region is identified by a binary prefix, and this shows the average |
| 76 | prefix length across all regions in the schedule. Longer prefixes indicate the |
| 77 | keyspace is divided into more regions (because there are more DHT servers in the |
| 78 | swarm to distribute records across). |
| 79 | |
| 80 | ### Next region prefix |
| 81 | |
| 82 | Keyspace prefix of the next region to be reprovided. |
| 83 | |
| 84 | ### Next region reprovide |
| 85 | |
| 86 | When the next region is scheduled to be reprovided. |
| 87 | |
| 88 | ## Timings |
| 89 | |
| 90 | ### Uptime |
| 91 | |
| 92 | How long the provide system has been running since Kubo started, along with the |
| 93 | start timestamp. |
| 94 | |
| 95 | ### Current time offset |
| 96 | |
| 97 | Elapsed time in the current reprovide cycle, showing cycle progress (e.g., '11h' |
| 98 | means 11 hours into a 22-hour cycle, roughly halfway through). |
| 99 | |
| 100 | ### Cycle started |
| 101 | |
| 102 | When the current reprovide cycle began. |
| 103 | |
| 104 | ### Reprovide interval |
| 105 | |
| 106 | How often each CID is reprovided (the complete cycle duration). |
| 107 | |
| 108 | ## Network |
| 109 | |
| 110 | ### Avg record holders |
| 111 | |
| 112 | Average number of provider records successfully sent for each CID to distinct |
| 113 | DHT servers. In practice, this is often lower than the [replication |
| 114 | factor](#replication-factor) due to unreachable peers or timeouts. Matching the |
| 115 | replication factor would indicate all DHT servers are reachable. |
| 116 | |
| 117 | Note: this counts successful sends; some DHT servers may have gone offline |
| 118 | afterward, so actual availability may be lower. |
| 119 | |
| 120 | ### Peers swept |
| 121 | |
| 122 | Number of DHT servers to which we tried to send provider records in the last |
| 123 | reprovide cycle (sweep). Excludes peers contacted during initial provides or |
| 124 | DHT lookups. |
| 125 | |
| 126 | ### Full keyspace coverage |
| 127 | |
| 128 | Whether provider records were sent to all DHT servers in the swarm during the |
| 129 | last reprovide cycle. If true, [peers swept](#peers-swept) approximates the |
| 130 | total DHT swarm size over the last [reprovide interval](#reprovide-interval). |
| 131 | |
| 132 | ### Reachable peers |
| 133 | |
| 134 | Number and percentage of peers to which we successfully sent all provider |
| 135 | records assigned to them during the last reprovide cycle. |
| 136 | |
| 137 | ### Avg region size |
| 138 | |
| 139 | Average number of DHT servers per keyspace region. |
| 140 | |
| 141 | ### Replication factor |
| 142 | |
| 143 | Target number of DHT servers to receive each provider record. |
| 144 | |
| 145 | ## Operations |
| 146 | |
| 147 | ### Ongoing provides |
| 148 | |
| 149 | Number of CIDs and regions currently being provided for the first time. More |
| 150 | CIDs than regions indicates efficient batching. Each region provide uses a |
| 151 | [burst |
| 152 | worker](./config.md#providedhtdedicatedburstworkers). |
| 153 | |
| 154 | ### Ongoing reprovides |
| 155 | |
| 156 | Number of CIDs and regions currently being reprovided. Each region reprovide |
| 157 | uses a [periodic |
| 158 | worker](./config.md#providedhtdedicatedperiodicworkers). |
| 159 | |
| 160 | ### Total CIDs provided |
| 161 | |
| 162 | Total number of provide operations since node startup (includes both provides |
| 163 | and reprovides). |
| 164 | |
| 165 | ### Total records provided |
| 166 | |
| 167 | Total provider records successfully sent to DHT servers since startup (includes |
| 168 | reprovides). |
| 169 | |
| 170 | ### Total provide errors |
| 171 | |
| 172 | Number of failed region provide/reprovide operations since startup. Failed |
| 173 | regions are automatically retried unless the node is offline. |
| 174 | |
| 175 | ### CIDs provided/min/worker |
| 176 | |
| 177 | Average rate of initial provides per minute per worker during the last |
| 178 | reprovide cycle (excludes reprovides). Each worker handles one keyspace region |
| 179 | at a time, providing all CIDs in that region. This measures the throughput of a |
| 180 | single worker only. |
| 181 | |
| 182 | To estimate total system provide throughput, multiply by the number of active |
| 183 | burst workers shown in [Workers stats](#workers-stats) (Burst > Active). |
| 184 | |
| 185 | Note: This rate only counts active time when initial provides are being |
| 186 | processed. If workers are idle, actual throughput may be lower. |
| 187 | |
| 188 | ### CIDs reprovided/min/worker |
| 189 | |
| 190 | Average rate of reprovides per minute per worker during the last reprovide |
| 191 | cycle (excludes initial provides). Each worker handles one keyspace region at a |
| 192 | time, reproviding all CIDs in that region. This measures the throughput of a |
| 193 | single worker only. |
| 194 | |
| 195 | To estimate total system reprovide throughput, multiply by the number of active |
| 196 | periodic workers shown in [Workers stats](#workers-stats) (Periodic > Active). |
| 197 | |
| 198 | Example: If this shows 100 CIDs/min and you have 10 active periodic workers, |
| 199 | your total reprovide throughput is approximately 1,000 CIDs/min. |
| 200 | |
| 201 | Note: This rate only counts active time when regions are being reprovided. If |
| 202 | workers are idle due to network issues or queue exhaustion, actual throughput |
| 203 | may be lower. |
| 204 | |
| 205 | ### Region reprovide duration |
| 206 | |
| 207 | Average time to reprovide all CIDs in a region during the last cycle. |
| 208 | |
| 209 | ### Avg CIDs/reprovide |
| 210 | |
| 211 | Average number of CIDs per region during the last reprovide cycle. |
| 212 | |
| 213 | This measures the average size of a region (how many CIDs are batched together), |
| 214 | not a throughput rate. Do NOT multiply this by worker count. |
| 215 | |
| 216 | Combined with [Region reprovide duration](#region-reprovide-duration), this |
| 217 | helps estimate per-worker throughput: dividing Avg CIDs/reprovide by Region |
| 218 | reprovide duration gives CIDs/min/worker. |
| 219 | |
| 220 | ### Regions reprovided (last cycle) |
| 221 | |
| 222 | Number of regions reprovided in the last cycle. |
| 223 | |
| 224 | > [!NOTE] |
| 225 | > (⚠️ 0.39 limitation) If this shows 1 region while using |
| 226 | > [`Routing.AcceleratedDHTClient`](./config.md#routingaccelerateddhtclient), sweep mode lost |
| 227 | > efficiency gains. Consider disabling the accelerated client. See [caveat 4](./config.md#routingaccelerateddhtclient). |
| 228 | |
| 229 | ## Workers |
| 230 | |
| 231 | ### Active workers |
| 232 | |
| 233 | Number of workers currently processing provide or reprovide operations. |
| 234 | |
| 235 | ### Free workers |
| 236 | |
| 237 | Number of idle workers not reserved for periodic or burst tasks. |
| 238 | |
| 239 | ### Workers stats |
| 240 | |
| 241 | Breakdown of worker status by type (periodic for scheduled reprovides, burst for |
| 242 | initial provides). For each type: |
| 243 | |
| 244 | - **Active**: Currently processing operations (use this count when calculating total throughput from per-worker rates) |
| 245 | - **Dedicated**: Reserved for this type |
| 246 | - **Available**: Idle dedicated workers + [free workers](#free-workers) |
| 247 | - **Queued**: 0 or 1 (workers acquired only when needed) |
| 248 | |
| 249 | The number of active workers determines your total system throughput. For |
| 250 | example, if you have 10 active periodic workers, multiply |
| 251 | [CIDs reprovided/min/worker](#cids-reprovidedminworker) by 10 to estimate total |
| 252 | reprovide throughput. |
| 253 | |
| 254 | See [provide queue](#provide-queue) and [reprovide queue](#reprovide-queue) for |
| 255 | regions waiting to be processed. |
| 256 | |
| 257 | ### Max connections/worker |
| 258 | |
| 259 | Maximum concurrent DHT server connections per worker when sending provider |
| 260 | records for a region. |
| 261 | |
| 262 | ## Capacity Planning |
| 263 | |
| 264 | ### Estimating if your system can keep up with the reprovide schedule |
| 265 | |
| 266 | To check if your provide system has sufficient capacity: |
| 267 | |
| 268 | 1. Calculate required throughput: |
| 269 | - Required CIDs/min = [CIDs scheduled](#cids-scheduled) / ([Reprovide interval](#reprovide-interval) in minutes) |
| 270 | - Example: 67M CIDs / (22 hours × 60 min) = 50,758 CIDs/min needed |
| 271 | |
| 272 | 2. Calculate actual throughput: |
| 273 | - Actual CIDs/min = [CIDs reprovided/min/worker](#cids-reprovidedminworker) × Active periodic workers |
| 274 | - Example: 100 CIDs/min/worker × 256 active workers = 25,600 CIDs/min |
| 275 | |
| 276 | 3. Compare: |
| 277 | - If actual < required: System is underprovisioned, increase [MaxWorkers](./config.md#providedhtmaxworkers) or [DedicatedPeriodicWorkers](./config.md#providedhtdedicatedperiodicworkers) |
| 278 | - If actual > required: System has excess capacity |
| 279 | - If [Reprovide queue](#reprovide-queue) is growing: System is falling behind |
| 280 | |
| 281 | ### Understanding worker utilization |
| 282 | |
| 283 | - High active workers with growing reprovide queue: Need more workers or network connectivity is limiting throughput |
| 284 | - Low active workers with non-empty reprovide queue: Workers may be waiting for network or DHT operations |
| 285 | - Check [Reachable peers](#reachable-peers) to diagnose network connectivity issues |
| 286 | - (⚠️ 0.39 limitation) If [Regions scheduled](#regions-scheduled) shows 1 while using |
| 287 | [`Routing.AcceleratedDHTClient`](./config.md#routingaccelerateddhtclient), consider disabling |
| 288 | the accelerated client to restore sweep efficiency. See [caveat 4](./config.md#routingaccelerateddhtclient). |
| 289 | |
| 290 | ## See Also |
| 291 | |
| 292 | - [Provide configuration reference](./config.md#provide) |
| 293 | - [Provide metrics for Prometheus](./metrics.md#provide) |