| 1 | # Delegated Routing Notes |
| 2 | |
| 3 | - Status Date: 2025-12 |
| 4 | |
| 5 | > [!IMPORTANT] |
| 6 | > Most users are best served by setting delegated HTTP router URLs in [`Routing.DelegatedRouters`](https://github.com/ipfs/kubo/blob/master/docs/config.md#routingdelegatedrouters) and `Routing.Type` to `auto` or `autoclient`, rather than using custom routing with `Routing.Routers` and `Routing.Methods` directly. |
| 7 | > |
| 8 | > The rest of this documentation describes experimental features intended only for researchers and advanced users. |
| 9 | |
| 10 | ---- |
| 11 | |
| 12 | # Custom Multi-Router Configuration (Experimental) |
| 13 | |
| 14 | - Start Date: 2022-08-15 |
| 15 | - Related Issues: |
| 16 | - https://github.com/ipfs/kubo/issues/9188 |
| 17 | - https://github.com/ipfs/kubo/issues/9079 |
| 18 | - https://github.com/ipfs/kubo/pull/9877 |
| 19 | |
| 20 | > [!CAUTION] |
| 21 | > **`Routing.Type=custom` with `Routing.Routers` and `Routing.Methods` is EXPERIMENTAL.** |
| 22 | > |
| 23 | > This feature is provided for **research and testing purposes only**. It is **not suitable for production use**. |
| 24 | > |
| 25 | > - The configuration format and behavior may change without notice between Kubo releases. |
| 26 | > - Bugs and regressions affecting custom routing may not be prioritized or fixed promptly. |
| 27 | > - HTTP-only routing configurations (without DHT) cannot reliably provide content to the network (👉️ see [Limitations](#limitations) below). |
| 28 | > |
| 29 | > **For production deployments**, use `Routing.Type=auto` (default) or `Routing.Type=autoclient` with [`Routing.DelegatedRouters`](https://github.com/ipfs/kubo/blob/master/docs/config.md#routingdelegatedrouters). |
| 30 | |
| 31 | ## Motivation |
| 32 | |
| 33 | The actual routing implementation is not enough. Some users need to have more options when configuring the routing system. The new implementations should be able to: |
| 34 | |
| 35 | - [x] Be user-friendly and easy enough to configure, but also versatile |
| 36 | - [x] Configurable Router execution order |
| 37 | - [x] Delay some of the Router methods execution when they will be executed on parallel |
| 38 | - [x] Configure which method of a giving router will be used |
| 39 | - [x] Mark some router methods as mandatory to make the execution fails if that method fails |
| 40 | |
| 41 | ## Detailed design |
| 42 | |
| 43 | ### Configuration file description |
| 44 | |
| 45 | The `Routing` configuration section will contain the following keys: |
| 46 | |
| 47 | #### Type |
| 48 | |
| 49 | `Type` will be still in use to avoid complexity for the user that only wants to use Kubo with the default behavior. We are going to add a new type, `custom`, that will use the new router systems. `none` type will deactivate **all** routers, default dht and delegated ones. |
| 50 | |
| 51 | #### Routers |
| 52 | |
| 53 | `Routers` will be a key-value list of routers that will be available to use. The key is the router name and the value is all the needed configurations for that router. the `Type` will define the routing kind. The main router types will be `http` and `dht`, but we will implement two special routers used to execute a set of routers in parallel or sequentially: `parallel` router and `sequential` router. |
| 54 | |
| 55 | Depending on the routing type, it will use different parameters: |
| 56 | |
| 57 | ##### HTTP |
| 58 | |
| 59 | Params: |
| 60 | |
| 61 | - `"Endpoint"`: URL of HTTP server with endpoints that implement [Delegated Routing V1 HTTP API](https://specs.ipfs.tech/routing/http-routing-v1/) protocol. |
| 62 | |
| 63 | ##### Amino DHT |
| 64 | |
| 65 | Params: |
| 66 | - `"Mode"`: Mode used by the Amino DHT. Possible values: "server", "client", "auto" |
| 67 | - `"AcceleratedDHTClient"`: Set to `true` if you want to use the experimentalDHT. |
| 68 | - `"PublicIPNetwork"`: Set to `true` to create a `WAN` Amino DHT. Set to `false` to create a `LAN` DHT. |
| 69 | |
| 70 | ##### Parallel |
| 71 | |
| 72 | Params: |
| 73 | - `Routers`: A list of routers that will be executed in parallel: |
| 74 | - `Name:string`: Name of the router. It should be one of the previously added to `Routers` list. |
| 75 | - `Timeout:duration`: Local timeout. It accepts strings compatible with Go `time.ParseDuration(string)`. Time will start counting when this specific router is called, and it will stop when the router returns, or we reach the specified timeout. |
| 76 | - `ExecuteAfter:duration`: Providing this param will delay the execution of that router at the specified time. It accepts strings compatible with Go `time.ParseDuration(string)`. |
| 77 | - `IgnoreErrors:bool`: It will specify if that router should be ignored if an error occurred. |
| 78 | - `Timeout:duration`: Global timeout. It accepts strings compatible with Go `time.ParseDuration(string)`. |
| 79 | ##### Sequential |
| 80 | |
| 81 | Params: |
| 82 | - `Routers`: A list of routers that will be executed in order: |
| 83 | - `Name:string`: Name of the router. It should be one of the previously added to `Routers` list. |
| 84 | - `Timeout:duration`: Local timeout. It accepts strings compatible with Go `time.ParseDuration(string)`. Time will start counting when this specific router is called, and it will stop when the router returns, or we reach the specified timeout. |
| 85 | - `IgnoreErrors:bool`: It will specify if that router should be ignored if an error occurred. |
| 86 | - `Timeout:duration`: Global timeout. It accepts strings compatible with Go `time.ParseDuration(string)`. |
| 87 | #### Methods |
| 88 | |
| 89 | `Methods:map` will define which routers will be executed per method. The key will be the name of the method: `"provide"`, `"find-providers"`, `"find-peers"`, `"put-ipns"`, `"get-ipns"`. All methods must be added to the list. This will make configuration discoverable giving good errors to the user if a method is missing. |
| 90 | |
| 91 | The value will contain: |
| 92 | - `RouterName:string`: Name of the router. It should be one of the previously added to `Routers` list. |
| 93 | |
| 94 | #### Configuration file example: |
| 95 | |
| 96 | ```json |
| 97 | "Routing": { |
| 98 | "Type": "custom", |
| 99 | "Routers": { |
| 100 | "http-delegated": { |
| 101 | "Type": "http", |
| 102 | "Parameters": { |
| 103 | "Endpoint": "https://delegated-ipfs.dev" // /routing/v1 (https://specs.ipfs.tech/routing/http-routing-v1/) |
| 104 | } |
| 105 | }, |
| 106 | "dht-lan": { |
| 107 | "Type": "dht", |
| 108 | "Parameters": { |
| 109 | "Mode": "server", |
| 110 | "PublicIPNetwork": false, |
| 111 | "AcceleratedDHTClient": false |
| 112 | } |
| 113 | }, |
| 114 | "dht-wan": { |
| 115 | "Type": "dht", |
| 116 | "Parameters": { |
| 117 | "Mode": "auto", |
| 118 | "PublicIPNetwork": true, |
| 119 | "AcceleratedDHTClient": false |
| 120 | } |
| 121 | }, |
| 122 | "find-providers-router": { |
| 123 | "Type": "parallel", |
| 124 | "Parameters": { |
| 125 | "Routers": [ |
| 126 | { |
| 127 | "RouterName": "dht-lan", |
| 128 | "IgnoreErrors": true |
| 129 | }, |
| 130 | { |
| 131 | "RouterName": "dht-wan" |
| 132 | }, |
| 133 | { |
| 134 | "RouterName": "http-delegated" |
| 135 | } |
| 136 | ] |
| 137 | } |
| 138 | }, |
| 139 | "provide-router": { |
| 140 | "Type": "parallel", |
| 141 | "Parameters": { |
| 142 | "Routers": [ |
| 143 | { |
| 144 | "RouterName": "dht-lan", |
| 145 | "IgnoreErrors": true |
| 146 | }, |
| 147 | { |
| 148 | "RouterName": "dht-wan", |
| 149 | "ExecuteAfter": "100ms", |
| 150 | "Timeout": "100ms" |
| 151 | }, |
| 152 | { |
| 153 | "RouterName": "http-delegated", |
| 154 | "ExecuteAfter": "100ms" |
| 155 | } |
| 156 | ] |
| 157 | } |
| 158 | }, |
| 159 | "get-ipns-router": { |
| 160 | "Type": "sequential", |
| 161 | "Parameters": { |
| 162 | "Routers": [ |
| 163 | { |
| 164 | "RouterName": "dht-lan", |
| 165 | "IgnoreErrors": true |
| 166 | }, |
| 167 | { |
| 168 | "RouterName": "dht-wan", |
| 169 | "Timeout": "300ms" |
| 170 | }, |
| 171 | { |
| 172 | "RouterName": "http-delegated", |
| 173 | "Timeout": "300ms" |
| 174 | } |
| 175 | ] |
| 176 | } |
| 177 | }, |
| 178 | "put-ipns-router": { |
| 179 | "Type": "parallel", |
| 180 | "Parameters": { |
| 181 | "Routers": [ |
| 182 | { |
| 183 | "RouterName": "dht-lan" |
| 184 | }, |
| 185 | { |
| 186 | "RouterName": "dht-wan" |
| 187 | }, |
| 188 | { |
| 189 | "RouterName": "http-delegated" |
| 190 | } |
| 191 | ] |
| 192 | } |
| 193 | } |
| 194 | }, |
| 195 | "Methods": { |
| 196 | "find-providers": { |
| 197 | "RouterName": "find-providers-router" |
| 198 | }, |
| 199 | "provide": { |
| 200 | "RouterName": "provide-router" |
| 201 | }, |
| 202 | "get-ipns": { |
| 203 | "RouterName": "get-ipns-router" |
| 204 | }, |
| 205 | "put-ipns": { |
| 206 | "RouterName": "put-ipns-router" |
| 207 | } |
| 208 | } |
| 209 | } |
| 210 | ``` |
| 211 | |
| 212 | ### Error cases |
| 213 | - If any of the routers fails, the output will be an error by default. |
| 214 | - You can use `IgnoreErrors:true` to ignore errors for a specific router output |
| 215 | - To avoid any error at the output, you must ignore all router errors. |
| 216 | |
| 217 | ### Implementation Details |
| 218 | |
| 219 | #### Methods |
| 220 | |
| 221 | All routers must implement the `routing.Routing` interface: |
| 222 | |
| 223 | ```go= |
| 224 | type Routing interface { |
| 225 | ContentRouting |
| 226 | PeerRouting |
| 227 | ValueStore |
| 228 | |
| 229 | Bootstrap(context.Context) error |
| 230 | } |
| 231 | ``` |
| 232 | |
| 233 | All methods involved: |
| 234 | |
| 235 | ```go= |
| 236 | type Routing interface { |
| 237 | Provide(context.Context, cid.Cid, bool) error |
| 238 | FindProvidersAsync(context.Context, cid.Cid, int) <-chan peer.AddrInfo |
| 239 | |
| 240 | FindPeer(context.Context, peer.ID) (peer.AddrInfo, error) |
| 241 | |
| 242 | PutValue(context.Context, string, []byte, ...Option) error |
| 243 | GetValue(context.Context, string, ...Option) ([]byte, error) |
| 244 | SearchValue(context.Context, string, ...Option) (<-chan []byte, error) |
| 245 | |
| 246 | Bootstrap(context.Context) error |
| 247 | } |
| 248 | ``` |
| 249 | We can configure which methods will be used per routing implementation. Methods names used in the configuration file will be: |
| 250 | |
| 251 | - `Provide`: `"provide"` |
| 252 | - `FindProvidersAsync`: `"find-providers"` |
| 253 | - `FindPeer`: `"find-peers"` |
| 254 | - `PutValue`: `"put-ipns"` |
| 255 | - `GetValue`, `SearchValue`: `"get-ipns"` |
| 256 | - `Bootstrap`: It will be always executed when needed. |
| 257 | |
| 258 | #### Routers |
| 259 | |
| 260 | We need to implement the `parallel` and `sequential` routers and stop using `routinghelpers.Tiered` router implementation. |
| 261 | |
| 262 | Add cycle detection to avoid to user some headaches. |
| 263 | |
| 264 | Also we need to implement an internal router, that will define the router used per method. |
| 265 | |
| 266 | #### Other considerations |
| 267 | |
| 268 | - We need to refactor how DHT routers are created to be able to use and add any amount of custom DHT routers. |
| 269 | - We need to add a new `custom` router type to be able to use the new routing system. |
| 270 | - Bitswap WANT broadcasting is not included on this document, but it can be added in next iterations. |
| 271 | - This document will live in docs/design-notes for historical reasons and future reference. |
| 272 | |
| 273 | ## Test fixtures |
| 274 | |
| 275 | As test fixtures we can add different use cases here and see how the configuration will look like. |
| 276 | |
| 277 | ### Mimic previous dual DHT config |
| 278 | |
| 279 | ```json |
| 280 | "Routing": { |
| 281 | "Type": "custom", |
| 282 | "Routers": { |
| 283 | "dht-lan": { |
| 284 | "Type": "dht", |
| 285 | "Parameters": { |
| 286 | "Mode": "server", |
| 287 | "PublicIPNetwork": false |
| 288 | } |
| 289 | }, |
| 290 | "dht-wan": { |
| 291 | "Type": "dht", |
| 292 | "Parameters": { |
| 293 | "Mode": "auto", |
| 294 | "PublicIPNetwork": true |
| 295 | } |
| 296 | }, |
| 297 | "parallel-dht-strict": { |
| 298 | "Type": "parallel", |
| 299 | "Parameters": { |
| 300 | "Routers": [ |
| 301 | { |
| 302 | "RouterName": "dht-lan" |
| 303 | }, |
| 304 | { |
| 305 | "RouterName": "dht-wan" |
| 306 | } |
| 307 | ] |
| 308 | } |
| 309 | }, |
| 310 | "parallel-dht": { |
| 311 | "Type": "parallel", |
| 312 | "Parameters": { |
| 313 | "Routers": [ |
| 314 | { |
| 315 | "RouterName": "dht-lan", |
| 316 | "IgnoreError": true |
| 317 | }, |
| 318 | { |
| 319 | "RouterName": "dht-wan" |
| 320 | } |
| 321 | ] |
| 322 | } |
| 323 | } |
| 324 | }, |
| 325 | "Methods": { |
| 326 | "provide": { |
| 327 | "RouterName": "dht-wan" |
| 328 | }, |
| 329 | "find-providers": { |
| 330 | "RouterName": "parallel-dht-strict" |
| 331 | }, |
| 332 | "find-peers": { |
| 333 | "RouterName": "parallel-dht-strict" |
| 334 | }, |
| 335 | "get-ipns": { |
| 336 | "RouterName": "parallel-dht" |
| 337 | }, |
| 338 | "put-ipns": { |
| 339 | "RouterName": "parallel-dht" |
| 340 | } |
| 341 | } |
| 342 | } |
| 343 | ``` |
| 344 | |
| 345 | ### Compatibility |
| 346 | |
| 347 | ~~We need to create a config migration using [fs-repo-migrations](https://github.com/ipfs/fs-repo-migrations). We should remove the `Routing.Type` param and add the configuration specified [previously](#Mimic-previous-dual-DHT-config).~~ |
| 348 | |
| 349 | We don't need to create any config migration! To avoid to the users the hassle of understanding how the new routing system works, we are going to keep the old behavior. We will add the Type `custom` to make available the new Routing system. |
| 350 | |
| 351 | ### Security |
| 352 | |
| 353 | No new security implications or considerations were found. |
| 354 | |
| 355 | ### Alternatives |
| 356 | |
| 357 | I got ideas from all of the following links to create this design document: |
| 358 | |
| 359 | - https://github.com/ipfs/kubo/issues/9079#issuecomment-1211288268 |
| 360 | - https://github.com/ipfs/kubo/issues/9157 |
| 361 | - https://github.com/ipfs/kubo/issues/9079#issuecomment-1205000253 |
| 362 | - https://www.notion.so/pl-strflt/Delegated-Routing-Thoughts-very-very-WIP-0543bc51b1bd4d63a061b0f28e195d38 |
| 363 | - https://gist.github.com/guseggert/effa027ff4cbadd7f67598efb6704d12 |
| 364 | |
| 365 | ### Limitations |
| 366 | |
| 367 | #### HTTP-only routing cannot reliably provide content |
| 368 | |
| 369 | Configurations that use only HTTP routers (without any DHT router) are unable to reliably announce content (provider records) to the network. |
| 370 | |
| 371 | This limitation exists because: |
| 372 | |
| 373 | 1. **No standardized HTTP API for providing**: The [Routing V1 HTTP API](https://specs.ipfs.tech/routing/http-routing-v1/) spec only defines read operations (`GET /routing/v1/providers/{cid}`). The write operation (`PUT /routing/v1/providers`) was never standardized. |
| 374 | |
| 375 | 2. **Legacy experimental API**: The only available HTTP providing mechanism is an undocumented `PUT /routing/v1/providers` request format called `ProvideBitswap`, which is a historical experiment. See [IPIP-526](https://github.com/ipfs/specs/pull/526) for ongoing discussion about formalizing HTTP-based provider announcements. |
| 376 | |
| 377 | 3. **Provider system integration**: Kubo's default provider system (`Provide.DHT.SweepEnabled=true` since v0.38) is designed for DHT-based providing. When no DHT is configured, the provider system may silently skip HTTP routers or behave unexpectedly. |
| 378 | |
| 379 | **Workarounds for testing:** |
| 380 | |
| 381 | If you need to test HTTP providing, you can try: |
| 382 | |
| 383 | - Setting `Provide.DHT.SweepEnabled=false` to use the legacy provider system |
| 384 | - Including at least one DHT router in your custom configuration alongside HTTP routers |
| 385 | |
| 386 | These workarounds are not guaranteed to work across Kubo versions and should not be relied upon for production use. |
| 387 | |
| 388 | ### Copyright |
| 389 | |
| 390 | Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). |