| 1 | # Experimental features of Kubo |
| 2 | |
| 3 | This document contains a list of experimental features in Kubo. |
| 4 | These features, commands, and APIs aren't mature, and you shouldn't rely on them. |
| 5 | Once they reach maturity, there's going to be mention in the changelog and |
| 6 | release posts. If they don't reach maturity, the same applies, and their code is |
| 7 | removed. |
| 8 | |
| 9 | Subscribe to https://github.com/ipfs/kubo/issues/3397 to get updates. |
| 10 | |
| 11 | When you add a new experimental feature to kubo or change an experimental |
| 12 | feature, you MUST please make a PR updating this document, and link the PR in |
| 13 | the above issue. |
| 14 | |
| 15 | - [Raw leaves for unixfs files](#raw-leaves-for-unixfs-files) |
| 16 | - [ipfs filestore](#ipfs-filestore) |
| 17 | - [ipfs urlstore](#ipfs-urlstore) |
| 18 | - [Private Networks](#private-networks) |
| 19 | - [ipfs p2p](#ipfs-p2p) |
| 20 | - [p2p http proxy](#p2p-http-proxy) |
| 21 | - [FUSE](#fuse) |
| 22 | - [Plugins](#plugins) |
| 23 | - [Directory Sharding / HAMT](#directory-sharding--hamt) |
| 24 | - [IPNS PubSub](#ipns-pubsub) |
| 25 | - [AutoRelay](#autorelay) |
| 26 | - [Strategic Providing](#strategic-providing) |
| 27 | - [Graphsync](#graphsync) |
| 28 | - [Noise](#noise) |
| 29 | - [Optimistic Provide](#optimistic-provide) |
| 30 | - [HTTP Gateway over Libp2p](#http-gateway-over-libp2p) |
| 31 | |
| 32 | --- |
| 33 | |
| 34 | ## Raw Leaves for unixfs files |
| 35 | |
| 36 | Allows files to be added with no formatting in the leaf nodes of the graph. |
| 37 | |
| 38 | ### State |
| 39 | |
| 40 | Stable but not used by default. |
| 41 | |
| 42 | ### In Version |
| 43 | |
| 44 | 0.4.5 |
| 45 | |
| 46 | ### How to enable |
| 47 | |
| 48 | Use `--raw-leaves` flag when calling `ipfs add`. This will save some space when adding files. |
| 49 | |
| 50 | ### Road to being a real feature |
| 51 | |
| 52 | Enabling this feature _by default_ will change the CIDs (hashes) of all newly imported files and will prevent newly imported files from deduplicating against previously imported files. While we do intend on enabling this by default, we plan on doing so once we have a large batch of "hash-changing" features we can enable all at once. |
| 53 | |
| 54 | ## ipfs filestore |
| 55 | |
| 56 | Allows files to be added without duplicating the space they take up on disk. |
| 57 | |
| 58 | ### State |
| 59 | |
| 60 | Experimental. |
| 61 | |
| 62 | ### In Version |
| 63 | |
| 64 | 0.4.7 |
| 65 | |
| 66 | ### How to enable |
| 67 | |
| 68 | > [!WARNING] |
| 69 | > **SECURITY CONSIDERATION** |
| 70 | > |
| 71 | > This feature provides the IPFS [`add` command](https://docs.ipfs.tech/reference/kubo/cli/#ipfs-add) with access to |
| 72 | > the local filesystem. Consequently, any user with access to CLI or the HTTP [`/v0/add` RPC API](https://docs.ipfs.tech/reference/kubo/rpc/#api-v0-add) can read |
| 73 | > files from the local filesystem with the same permissions as the Kubo daemon. |
| 74 | > If you enable this, secure your RPC API using [`API.Authorizations`](https://github.com/ipfs/kubo/blob/master/docs/config.md#apiauthorizations) or custom auth middleware. |
| 75 | |
| 76 | Modify your ipfs config: |
| 77 | ``` |
| 78 | ipfs config --json Experimental.FilestoreEnabled true |
| 79 | ``` |
| 80 | |
| 81 | Then restart your IPFS node to reload your config. |
| 82 | |
| 83 | Finally, when adding files with ipfs add, pass the --nocopy flag to use the |
| 84 | filestore instead of copying the files into your local IPFS repo. |
| 85 | |
| 86 | ### Road to being a real feature |
| 87 | |
| 88 | - [ ] Needs more people to use and report on how well it works. |
| 89 | - [ ] Need to address error states and failure conditions |
| 90 | - [ ] cleanup of broken filesystem references (if file is deleted) |
| 91 | - [ ] tests that confirm ability to override preexisting filesystem links (allowing user to fix broken link) |
| 92 | - [ ] support for a single block having more than one sources in filesystem (blocks can be shared by unrelated files, and not be broken when some files are unpinned / gc'd) |
| 93 | - [ ] [other known issues](https://github.com/ipfs/kubo/issues/7161) |
| 94 | - [ ] Need to write docs on usage, advantages, disadvantages |
| 95 | - [ ] Need to merge utility commands to aid in maintenance and repair of filestore |
| 96 | |
| 97 | ## ipfs urlstore |
| 98 | |
| 99 | Allows ipfs to retrieve blocks contents via a URL instead of storing it in the datastore |
| 100 | |
| 101 | ### State |
| 102 | |
| 103 | Experimental. |
| 104 | |
| 105 | ### In Version |
| 106 | |
| 107 | v0.4.17 |
| 108 | |
| 109 | ### How to enable |
| 110 | |
| 111 | > [!WARNING] |
| 112 | > **SECURITY CONSIDERATION** |
| 113 | > |
| 114 | > This feature provides the IPFS [`add` CLI command](https://docs.ipfs.tech/reference/kubo/cli/#ipfs-add) with access to |
| 115 | > the local filesystem. Consequently, any user with access to the CLI or HTTP [`/v0/add` RPC API](https://docs.ipfs.tech/reference/kubo/rpc/#api-v0-add) can read |
| 116 | > files from the local filesystem with the same permissions as the Kubo daemon. |
| 117 | > If you enable this, secure your RPC API using [`API.Authorizations`](https://github.com/ipfs/kubo/blob/master/docs/config.md#apiauthorizations) or custom auth middleware. |
| 118 | |
| 119 | Modify your ipfs config: |
| 120 | ``` |
| 121 | ipfs config --json Experimental.UrlstoreEnabled true |
| 122 | ``` |
| 123 | |
| 124 | And then add a file at a specific URL using `ipfs urlstore add <url>` |
| 125 | |
| 126 | ### Road to being a real feature |
| 127 | - [ ] Needs more people to use and report on how well it works. |
| 128 | - [ ] Need to address error states and failure conditions |
| 129 | - [ ] cleanup of broken URL+range references (if URL starts returning 404 or error) |
| 130 | - [ ] tests that confirm ability to override preexisting URL+range links (allowing user to fix broken link) |
| 131 | - [ ] support for a single block having more than one URL+range (blocks can be shared by unrelated URLs) |
| 132 | - [ ] Need to write docs on usage, advantages, disadvantages |
| 133 | - [ ] Need to implement caching |
| 134 | - [ ] Need to add metrics to monitor performance |
| 135 | |
| 136 | ## Private Networks |
| 137 | |
| 138 | It allows ipfs to only connect to other peers who have a shared secret key. |
| 139 | |
| 140 | ### State |
| 141 | |
| 142 | Stable but not quite ready for prime-time. |
| 143 | |
| 144 | > [!WARNING] |
| 145 | > Limited to TCP transport, comes with overhead of double-encryption. See details below. |
| 146 | |
| 147 | ### In Version |
| 148 | |
| 149 | 0.4.7 |
| 150 | |
| 151 | ### How to enable |
| 152 | |
| 153 | Generate a pre-shared-key using [ipfs-swarm-key-gen](https://github.com/Kubuxu/go-ipfs-swarm-key-gen)): |
| 154 | ``` |
| 155 | go install github.com/Kubuxu/go-ipfs-swarm-key-gen/ipfs-swarm-key-gen@latest |
| 156 | ipfs-swarm-key-gen > ~/.ipfs/swarm.key |
| 157 | ``` |
| 158 | |
| 159 | To join a given private network, get the key file from someone in the network |
| 160 | and save it to `~/.ipfs/swarm.key` (If you are using a custom `$IPFS_PATH`, put |
| 161 | it in there instead). |
| 162 | |
| 163 | When using this feature, you will not be able to connect to the default bootstrap |
| 164 | nodes (Since we aren't part of your private network) so you will need to set up |
| 165 | your own bootstrap nodes. |
| 166 | |
| 167 | First, to prevent your node from even trying to connect to the default bootstrap nodes, run: |
| 168 | ```bash |
| 169 | ipfs bootstrap rm --all |
| 170 | ``` |
| 171 | |
| 172 | Then add your own bootstrap peers with: |
| 173 | ```bash |
| 174 | ipfs bootstrap add <multiaddr> |
| 175 | ``` |
| 176 | |
| 177 | For example: |
| 178 | ``` |
| 179 | ipfs bootstrap add /ip4/104.236.76.40/tcp/4001/p2p/QmSoLV4Bbm51jM9C4gDYZQ9Cy3U6aXMJDAbzgu2fzaDs64 |
| 180 | ``` |
| 181 | |
| 182 | Bootstrap nodes are no different from all other nodes in the network apart from |
| 183 | the function they serve. |
| 184 | |
| 185 | To be extra cautious, You can also set the `LIBP2P_FORCE_PNET` environment |
| 186 | variable to `1` to force the usage of private networks. If no private network is |
| 187 | configured, the daemon will fail to start. |
| 188 | |
| 189 | ### Road to being a real feature |
| 190 | |
| 191 | - [x] Needs more people to use and report on how well it works |
| 192 | - [ ] More documentation |
| 193 | - [ ] Improve / future proof libp2p support (see [libp2p/specs#489](https://github.com/libp2p/specs/issues/489)) |
| 194 | - [ ] Currently limited to TCP-only, and double-encrypts all data sent on TCP. This is slow. |
| 195 | - [ ] Does not work with QUIC: [go-libp2p#1432](https://github.com/libp2p/go-libp2p/issues/1432) |
| 196 | - [ ] Needs better tooling/UX |
| 197 | - [ ] Detect lack of peers when swarm key is present and prompt user to set up bootstrappers/peering |
| 198 | - [ ] ipfs-webui will not load unless blocks are present in private swarm. Detect it and prompt user to import CAR with webui. |
| 199 | |
| 200 | ## ipfs p2p |
| 201 | |
| 202 | Allows tunneling of TCP connections through libp2p streams, similar to SSH port |
| 203 | forwarding (`ssh -L`). |
| 204 | |
| 205 | ### State |
| 206 | |
| 207 | Experimental, will be stabilized in 0.6.0 |
| 208 | |
| 209 | ### In Version |
| 210 | |
| 211 | 0.4.10 |
| 212 | |
| 213 | ### How to enable |
| 214 | |
| 215 | > [!WARNING] |
| 216 | > **SECURITY CONSIDERATION** |
| 217 | > |
| 218 | > This feature provides CLI and HTTP RPC user with ability to set up port forwarding for all localhost and LAN ports. |
| 219 | > If you enable this and plan to expose CLI or HTTP RPC to other users or machines, |
| 220 | > secure RPC API using [`API.Authorizations`](https://github.com/ipfs/kubo/blob/master/docs/config.md#apiauthorizations) or custom auth middleware. |
| 221 | |
| 222 | ```sh |
| 223 | > ipfs config --json Experimental.Libp2pStreamMounting true |
| 224 | ``` |
| 225 | |
| 226 | ### How to use |
| 227 | |
| 228 | See [docs/p2p-tunnels.md](p2p-tunnels.md) for usage examples, foreground mode, |
| 229 | and systemd integration. |
| 230 | |
| 231 | ### Road to being a real feature |
| 232 | |
| 233 | - [x] More documentation |
| 234 | - [x] `ipfs p2p forward` mode |
| 235 | - [ ] Ability to define tunnels via JSON config, similar to [`Peering.Peers`](https://github.com/ipfs/kubo/blob/master/docs/config.md#peeringpeers), see [kubo#5460](https://github.com/ipfs/kubo/issues/5460) |
| 236 | |
| 237 | ## p2p http proxy |
| 238 | |
| 239 | Allows proxying of HTTP requests over p2p streams. This allows serving any standard HTTP app over p2p streams. |
| 240 | |
| 241 | ### State |
| 242 | |
| 243 | Experimental |
| 244 | |
| 245 | ### In Version |
| 246 | |
| 247 | 0.4.19 |
| 248 | |
| 249 | ### How to enable |
| 250 | |
| 251 | > [!WARNING] |
| 252 | > **SECURITY CONSIDERATION** |
| 253 | > |
| 254 | > This feature provides CLI and HTTP RPC user with ability to set up HTTP forwarding for all localhost and LAN ports. |
| 255 | > If you enable this and plan to expose CLI or HTTP RPC to other users or machines, |
| 256 | > secure RPC API using [`API.Authorizations`](https://github.com/ipfs/kubo/blob/master/docs/config.md#apiauthorizations) or custom auth middleware. |
| 257 | |
| 258 | The `p2p` command needs to be enabled in the config: |
| 259 | |
| 260 | ```sh |
| 261 | > ipfs config --json Experimental.Libp2pStreamMounting true |
| 262 | ``` |
| 263 | |
| 264 | On the client, the p2p HTTP proxy needs to be enabled in the config: |
| 265 | |
| 266 | ```sh |
| 267 | > ipfs config --json Experimental.P2pHttpProxy true |
| 268 | ``` |
| 269 | |
| 270 | ### How to use |
| 271 | |
| 272 | **Netcat example:** |
| 273 | |
| 274 | First, pick a protocol name for your application. Think of the protocol name as |
| 275 | a port number, just significantly more user-friendly. In this example, we're |
| 276 | going to use `/http`. |
| 277 | |
| 278 | ***Setup:*** |
| 279 | |
| 280 | 1. A "server" node with peer ID `$SERVER_ID` |
| 281 | 2. A "client" node. |
| 282 | |
| 283 | ***On the "server" node:*** |
| 284 | |
| 285 | First, start your application and have it listen for TCP connections on |
| 286 | port `$APP_PORT`. |
| 287 | |
| 288 | Then, configure the p2p listener by running: |
| 289 | |
| 290 | ```sh |
| 291 | > ipfs p2p listen --allow-custom-protocol /http /ip4/127.0.0.1/tcp/$APP_PORT |
| 292 | ``` |
| 293 | |
| 294 | This will configure IPFS to forward all incoming `/http` streams to |
| 295 | `127.0.0.1:$APP_PORT` (opening a new connection to `127.0.0.1:$APP_PORT` per incoming stream. |
| 296 | |
| 297 | ***On the "client" node:*** |
| 298 | |
| 299 | Next, have your application make a http request to `127.0.0.1:8080/p2p/$SERVER_ID/http/$FORWARDED_PATH`. This |
| 300 | connection will be forwarded to the service running on `127.0.0.1:$APP_PORT` on |
| 301 | the remote machine (which needs to be a http server!) with path `$FORWARDED_PATH`. You can test it with netcat: |
| 302 | |
| 303 | ***On "server" node:*** |
| 304 | ```sh |
| 305 | > echo -e "HTTP/1.1 200\nContent-length: 11\n\nIPFS rocks!" | nc -l -p $APP_PORT |
| 306 | ``` |
| 307 | |
| 308 | ***On "client" node:*** |
| 309 | ```sh |
| 310 | > curl http://localhost:8080/p2p/$SERVER_ID/http/ |
| 311 | ``` |
| 312 | |
| 313 | You should now see the resulting HTTP response: IPFS rocks! |
| 314 | |
| 315 | ### Custom protocol names |
| 316 | |
| 317 | We also support the use of protocol names of the form /x/$NAME/http where $NAME doesn't contain any "/"'s |
| 318 | |
| 319 | ### Road to being a real feature |
| 320 | |
| 321 | - [ ] Needs p2p streams to graduate from experiments |
| 322 | - [ ] Needs more people to use and report on how well it works and fits use cases |
| 323 | - [ ] More documentation |
| 324 | - [ ] Need better integration with the subdomain gateway feature. |
| 325 | |
| 326 | ## FUSE |
| 327 | |
| 328 | FUSE makes it possible to mount `/ipfs`, `/ipns` and `/mfs` namespaces in your OS, |
| 329 | allowing arbitrary apps access to IPFS using standard filesystem operations. |
| 330 | |
| 331 | It is considered EXPERIMENTAL due to limited support on some platforms. |
| 332 | |
| 333 | See [fuse.md](./fuse.md) for setup instructions and details. |
| 334 | |
| 335 | ## Plugins |
| 336 | |
| 337 | ### In Version |
| 338 | 0.4.11 |
| 339 | |
| 340 | ### State |
| 341 | Experimental |
| 342 | |
| 343 | Plugins allow adding functionality without the need to recompile the daemon. |
| 344 | |
| 345 | ### Basic Usage: |
| 346 | |
| 347 | See [Plugin docs](./plugins.md) |
| 348 | |
| 349 | ### Road to being a real feature |
| 350 | |
| 351 | - [x] More plugins and plugin types |
| 352 | - [ ] A way to reliably build and distribute plugins. |
| 353 | - [ ] Better support for platforms other than Linux & MacOS |
| 354 | - [ ] Feedback on stability |
| 355 | |
| 356 | ## Directory Sharding / HAMT |
| 357 | |
| 358 | ### In Version |
| 359 | |
| 360 | - 0.4.8: |
| 361 | - Introduced `Experimental.ShardingEnabled` which enabled sharding globally. |
| 362 | - All-or-nothing, unnecessary sharding of small directories. |
| 363 | |
| 364 | - 0.11.0 : |
| 365 | - Removed support for `Experimental.ShardingEnabled` |
| 366 | - Replaced with automatic sharding based on the block size |
| 367 | |
| 368 | ### State |
| 369 | |
| 370 | Replaced by autosharding. |
| 371 | |
| 372 | The `Experimental.ShardingEnabled` config field is no longer used, please remove it from your configs. |
| 373 | |
| 374 | kubo now automatically shards when directory block is bigger than 256KB, ensuring every block is small enough to be exchanged with other peers |
| 375 | |
| 376 | ## IPNS pubsub |
| 377 | |
| 378 | Specification: [IPNS PubSub Router](https://specs.ipfs.tech/ipns/ipns-pubsub-router/) |
| 379 | |
| 380 | ### In Version |
| 381 | |
| 382 | 0.4.14 : |
| 383 | - Introduced |
| 384 | |
| 385 | 0.5.0 : |
| 386 | - No longer needs to use the DHT for the first resolution |
| 387 | - When discovering PubSub peers via the DHT, the DHT key is different from previous versions |
| 388 | - This leads to 0.5 IPNS pubsub peers and 0.4 IPNS pubsub peers not being able to find each other in the DHT |
| 389 | - Robustness improvements |
| 390 | |
| 391 | 0.11.0 : |
| 392 | - Can be enabled via `Ipns.UsePubsub` flag in config |
| 393 | |
| 394 | 0.40.0 : |
| 395 | - Persistent message sequence number validation to prevent message cycles |
| 396 | in large networks |
| 397 | |
| 398 | ### State |
| 399 | |
| 400 | Experimental, default-disabled. |
| 401 | |
| 402 | Utilizes pubsub for publishing IPNS records in real time. |
| 403 | |
| 404 | When it is enabled: |
| 405 | |
| 406 | - IPNS publishers push records to a name-specific pubsub topic, |
| 407 | in addition to publishing to the DHT. |
| 408 | - IPNS resolvers subscribe to the name-specific topic on first |
| 409 | resolution and receive subsequently published records through pubsub in real time. |
| 410 | This makes subsequent resolutions instant, as they are resolved through the local cache. |
| 411 | |
| 412 | Both the publisher and the resolver nodes need to have the feature enabled for it to work effectively. |
| 413 | |
| 414 | ### How to enable |
| 415 | |
| 416 | Run your daemon with the `--enable-namesys-pubsub` flag |
| 417 | or modify your ipfs config and restart the daemon: |
| 418 | ``` |
| 419 | ipfs config --json Ipns.UsePubsub true |
| 420 | ``` |
| 421 | |
| 422 | NOTE: |
| 423 | - This feature implicitly enables pubsub. |
| 424 | - Passing `--enable-namesys-pubsub` CLI flag overrides `Ipns.UsePubsub` config. |
| 425 | |
| 426 | ### Road to being a real feature |
| 427 | |
| 428 | - [ ] Needs more people to use and report on how well it works |
| 429 | |
| 430 | ## AutoRelay |
| 431 | |
| 432 | ### In Version |
| 433 | |
| 434 | - 0.4.19 : |
| 435 | - Introduced Circuit Relay v1 |
| 436 | - 0.11.0 : |
| 437 | - Deprecated v1 |
| 438 | - Introduced [Circuit Relay v2](https://github.com/libp2p/specs/blob/master/relay/circuit-v2.md) |
| 439 | |
| 440 | ### State |
| 441 | |
| 442 | Experimental, disabled by default. |
| 443 | |
| 444 | Automatically discovers relays and advertises relay addresses when the node is behind an impenetrable NAT. |
| 445 | |
| 446 | ### How to enable |
| 447 | |
| 448 | Modify your ipfs config: |
| 449 | |
| 450 | ``` |
| 451 | ipfs config --json Swarm.RelayClient.Enabled true |
| 452 | ``` |
| 453 | |
| 454 | ### Road to being a real feature |
| 455 | |
| 456 | - [ ] needs testing |
| 457 | - [ ] needs to be automatically enabled when AutoNAT detects node is behind an impenetrable NAT. |
| 458 | |
| 459 | |
| 460 | ## Strategic Providing |
| 461 | |
| 462 | ### State |
| 463 | |
| 464 | `Experimental.StrategicProviding` was removed in Kubo v0.35. |
| 465 | |
| 466 | Replaced by [`Provide.Enabled`](https://github.com/ipfs/kubo/blob/master/docs/config.md#provideenabled) and [`Provide.Strategy`](https://github.com/ipfs/kubo/blob/master/docs/config.md#providestrategy). |
| 467 | |
| 468 | ## GraphSync |
| 469 | |
| 470 | ### State |
| 471 | |
| 472 | Removed, no plans to reintegrate either as experimental or stable feature. |
| 473 | |
| 474 | [Trustless Gateway over Libp2p](#http-gateway-over-libp2p) should be easier to use for unixfs usecases and support basic wildcard car streams for non unixfs. |
| 475 | |
| 476 | See https://github.com/ipfs/kubo/pull/9747 for more information. |
| 477 | |
| 478 | ## Noise |
| 479 | |
| 480 | ### State |
| 481 | |
| 482 | Stable, enabled by default |
| 483 | |
| 484 | [Noise](https://github.com/libp2p/specs/tree/master/noise) libp2p transport based on the [Noise Protocol Framework](https://noiseprotocol.org/noise.html). While TLS remains the default transport in Kubo, Noise is easier to implement and is thus the "interop" transport between IPFS and libp2p implementations. |
| 485 | |
| 486 | ## Optimistic Provide |
| 487 | |
| 488 | ### In Version |
| 489 | |
| 490 | 0.20.0 |
| 491 | |
| 492 | ### State |
| 493 | |
| 494 | Experimental, disabled by default. |
| 495 | |
| 496 | When the Amino DHT client tries to store a provider in the DHT, it typically searches for the 20 peers that are closest to the |
| 497 | target key. However, this process can be time-consuming, as the search terminates only after no closer peers are found |
| 498 | among the three currently (during the query) known closest ones. In cases where these closest peers are slow to respond |
| 499 | (which often happens if they are located at the edge of the DHT network), the query gets blocked by the slowest peer. |
| 500 | |
| 501 | To address this issue, the `OptimisticProvide` feature can be enabled. This feature allows the client to estimate the |
| 502 | network size and determine how close a peer _likely_ needs to be to the target key to be within the 20 closest peers. |
| 503 | While searching for the closest peers in the DHT, the client will _optimistically_ store the provider record with peers |
| 504 | and abort the query completely when the set of currently known 20 closest peers are also _likely_ the actual 20 closest |
| 505 | ones. This heuristic approach can significantly speed up the process, resulting in a speed improvement of 2x to >10x. |
| 506 | |
| 507 | When it is enabled: |
| 508 | |
| 509 | - Amino DHT provide operations should complete much faster than with it disabled |
| 510 | - This can be tested with commands such as `ipfs provide once` |
| 511 | |
| 512 | **Tradeoffs** |
| 513 | |
| 514 | There are now the classic client, the accelerated DHT client, and optimistic provide that improve the provider process. |
| 515 | There are different trade-offs with all of them. The accelerated DHT client is still faster to provide large amounts |
| 516 | of provider records at the cost of high resource requirements. Optimistic provide doesn't have the high resource |
| 517 | requirements but might not choose optimal peers and is not as fast as the accelerated client, but still much faster |
| 518 | than the classic client. |
| 519 | |
| 520 | **Caveats:** |
| 521 | |
| 522 | 1. Providing optimistically requires a current network size estimation. This estimation is calculated through routing |
| 523 | table refresh queries and is only available after the daemon has been running for some time. If there is no network |
| 524 | size estimation available the client will transparently fall back to the classic approach. |
| 525 | 2. The chosen peers to store the provider records might not be the actual closest ones. Measurements showed that this |
| 526 | is not a problem. |
| 527 | 3. The optimistic provide process returns already after 15 out of the 20 provider records were stored with peers. The |
| 528 | reasoning here is that one out of the remaining 5 peers are very likely to time out and delay the whole process. To |
| 529 | limit the number of in-flight async requests there is the second `OptimisticProvideJobsPoolSize` setting. Currently, |
| 530 | this is set to 60. This means that at most 60 parallel background requests are allowed to be in-flight. If this |
| 531 | limit is exceeded optimistic provide will block until all 20 provider records are written. This is still 2x faster |
| 532 | than the classic approach but not as fast as returning early which yields >10x speed-ups. |
| 533 | 4. Since the in-flight background requests are likely to time out, they are not consuming many resources and the job |
| 534 | pool size could probably be much higher. |
| 535 | |
| 536 | For more information, see: |
| 537 | |
| 538 | - Project doc: https://protocollabs.notion.site/Optimistic-Provide-2c79745820fa45649d48de038516b814 |
| 539 | - go-libp2p-kad-dht: https://github.com/libp2p/go-libp2p-kad-dht/pull/783 |
| 540 | |
| 541 | ### Configuring |
| 542 | To enable: |
| 543 | |
| 544 | ``` |
| 545 | ipfs config --json Experimental.OptimisticProvide true |
| 546 | ``` |
| 547 | |
| 548 | If you want to change the `OptimisticProvideJobsPoolSize` setting from its default of 60: |
| 549 | |
| 550 | ``` |
| 551 | ipfs config --json Experimental.OptimisticProvideJobsPoolSize 120 |
| 552 | ``` |
| 553 | |
| 554 | ### Road to being a real feature |
| 555 | |
| 556 | - [ ] Needs more people to use and report on how well it works |
| 557 | - [ ] Should prove at least equivalent availability of provider records as the classic approach |
| 558 | |
| 559 | ## HTTP Gateway over Libp2p |
| 560 | |
| 561 | ### In Version |
| 562 | |
| 563 | 0.23.0 |
| 564 | |
| 565 | ### State |
| 566 | |
| 567 | Experimental, disabled by default. |
| 568 | |
| 569 | Enables serving a subset of the [IPFS HTTP Gateway](https://specs.ipfs.tech/http-gateways/) semantics over libp2p `/http/1.1` protocol. |
| 570 | |
| 571 | Notes: |
| 572 | - This feature only about serving verifiable gateway requests over libp2p: |
| 573 | - Deserialized responses are not supported. |
| 574 | - Only operate on `/ipfs` resources (no `/ipns` atm) |
| 575 | - Only support requests for `application/vnd.ipld.raw` and |
| 576 | `application/vnd.ipld.car` (from [Trustless Gateway Specification](https://specs.ipfs.tech/http-gateways/trustless-gateway/), |
| 577 | where data integrity can be verified). |
| 578 | - Only serve data that is already local to the node (i.e. similar to a |
| 579 | [`Gateway.NoFetch`](https://github.com/ipfs/kubo/blob/master/docs/config.md#gatewaynofetch)) |
| 580 | - While Kubo currently mounts the gateway API at the root (i.e. `/`) of the |
| 581 | libp2p `/http/1.1` protocol, that is subject to change. |
| 582 | - The way to reliably discover where a given HTTP protocol is mounted on a |
| 583 | libp2p endpoint is via the `.well-known/libp2p` resource specified in the |
| 584 | [http+libp2p specification](https://github.com/libp2p/specs/pull/508) |
| 585 | - The identifier of the protocol mount point under `/http/1.1` listener is |
| 586 | `/ipfs/gateway`, as noted in |
| 587 | [ipfs/specs#434](https://github.com/ipfs/specs/pull/434). |
| 588 | |
| 589 | ### How to enable |
| 590 | |
| 591 | Modify your ipfs config: |
| 592 | |
| 593 | ``` |
| 594 | ipfs config --json Experimental.GatewayOverLibp2p true |
| 595 | ``` |
| 596 | |
| 597 | ### Road to being a real feature |
| 598 | |
| 599 | - [ ] Needs more people to use and report on how well it works |
| 600 | - [ ] Needs UX work for exposing non-recursive "HTTP transport" (NoFetch) over both libp2p and plain TCP (and sharing the configuration) |
| 601 | - [ ] Needs a mechanism for HTTP handler to signal supported features ([IPIP-425](https://github.com/ipfs/specs/pull/425)) |
| 602 | - [ ] Needs an option for Kubo to detect peers that have it enabled and prefer HTTP transport before falling back to bitswap (and use CAR if peer supports dag-scope=entity from [IPIP-402](https://specs.ipfs.tech/ipips/ipip-0402/)) |
| 603 | |
| 604 | ## Accelerated DHT Client |
| 605 | |
| 606 | This feature now lives at [`Routing.AcceleratedDHTClient`](https://github.com/ipfs/kubo/blob/master/docs/config.md#routingaccelerateddhtclient). |
| 607 |