| 1 | # Datastore Configuration Options |
| 2 | |
| 3 | This document describes the different possible values for the `Datastore.Spec` |
| 4 | field in the ipfs configuration file. |
| 5 | |
| 6 | - [flatfs](#flatfs) |
| 7 | - [levelds](#levelds) |
| 8 | - [pebbleds](#pebbleds) |
| 9 | - [badgerds](#badgerds) |
| 10 | - [mount](#mount) |
| 11 | - [measure](#measure) |
| 12 | |
| 13 | ## flatfs |
| 14 | |
| 15 | Stores each key-value pair as a file on the filesystem. |
| 16 | |
| 17 | The shardFunc is prefixed with `/repo/flatfs/shard/v1` then followed by a descriptor of the sharding strategy. Some example values are: |
| 18 | - `/repo/flatfs/shard/v1/next-to-last/2` |
| 19 | - Shards on the two next-to-last base32 characters of the key (~1024 directories) |
| 20 | - `/repo/flatfs/shard/v1/next-to-last/3` |
| 21 | - Shards on the three next-to-last base32 characters of the key (~32,768 directories) |
| 22 | - `/repo/flatfs/shard/v1/prefix/2` |
| 23 | - Shards based on the two-character prefix of the key |
| 24 | |
| 25 | ```json |
| 26 | { |
| 27 | "type": "flatfs", |
| 28 | "path": "<relative path within repo for flatfs root>", |
| 29 | "shardFunc": "<a descriptor of the sharding scheme>", |
| 30 | "sync": true|false |
| 31 | } |
| 32 | ``` |
| 33 | |
| 34 | - `sync`: Flush every write to disk before continuing. Setting this to false is safe as kubo will automatically flush writes to disk before and after performing critical operations like pinning. However, you can set this to true to be extra-safe (at the cost of a slowdown when adding files). |
| 35 | |
| 36 | NOTE: flatfs must only be used as a block store (mounted at `/blocks`) as it only partially implements the datastore interface. You can mount flatfs for /blocks only using the mount datastore (described below). |
| 37 | |
| 38 | ### Choosing a `shardFunc` for large blockstores |
| 39 | |
| 40 | The `next-to-last/N` shard depth controls how many directories the blockstore |
| 41 | is spread across. Each shard becomes a single directory under `blocks/`, and |
| 42 | every block file lives directly inside its shard. The cost of any operation |
| 43 | that does a `readdir` or per-file `stat` on a shard scales with the number of |
| 44 | files in that shard. |
| 45 | |
| 46 | Two depths in common use: |
| 47 | |
| 48 | | `shardFunc` | Shard count | At 60M blocks | Notes | |
| 49 | |--------------------------|------------:|----------------:|---------------------------------------------| |
| 50 | | `next-to-last/2` | ~1,024 | ~58k files/dir | default; fine for small/medium nodes | |
| 51 | | `next-to-last/3` | ~32,768 | ~1.8k files/dir | recommended for large pinning/gateway nodes | |
| 52 | |
| 53 | For nodes expected to grow past a few million blocks (most pinning clusters, |
| 54 | public gateways, mirrors), prefer `next-to-last/3`. The deeper sharding keeps |
| 55 | per-directory file counts in a range modern filesystems handle well, and it |
| 56 | significantly reduces the per-operation cost of `Stat`, `readdir`, and bulk |
| 57 | enumeration (used by GC, [`Datastore.BloomFilterSize`](config.md#datastorebloomfiltersize) |
| 58 | rebuild on startup, and `Provide.Strategy=all` reprovide cycles). On nodes |
| 59 | backed by rotational disks the difference can be the gap between healthy |
| 60 | operation and IOPS-saturated iowait. |
| 61 | |
| 62 | The shard depth is fixed at `ipfs init` time. Kubo ships no in-place |
| 63 | re-sharding tool, so switching depth on an existing repo means exporting |
| 64 | and re-importing the blockstore. Pick conservatively for the expected |
| 65 | steady state of the node. |
| 66 | |
| 67 | ## levelds |
| 68 | |
| 69 | Uses a [leveldb](https://github.com/syndtr/goleveldb) database to store key-value |
| 70 | pairs via [go-ds-leveldb](https://github.com/ipfs/go-ds-leveldb). |
| 71 | |
| 72 | ```json |
| 73 | { |
| 74 | "type": "levelds", |
| 75 | "path": "<location of db inside repo>", |
| 76 | "compression": "none" | "snappy", |
| 77 | } |
| 78 | ``` |
| 79 | |
| 80 | > [!NOTE] |
| 81 | > LevelDB uses a log-structured merge-tree (LSM) storage engine. When keys are |
| 82 | > deleted, the data is not removed immediately. Instead, a tombstone marker is |
| 83 | > written, and the actual data is removed later by background compaction. |
| 84 | > |
| 85 | > LevelDB's compaction decides what to compact based on file counts (L0) and |
| 86 | > total level size (L1+), without considering how many tombstones a file |
| 87 | > contains. This means that after bulk deletions (such as pin removals or the |
| 88 | > periodic provider keystore sync), disk space may not be reclaimed promptly. |
| 89 | > The `datastore/` directory can grow significantly larger than the live data it |
| 90 | > holds, especially on long-running nodes with many CIDs. |
| 91 | > |
| 92 | > Unlike flatfs (which deletes files immediately) or pebble (which has |
| 93 | > tombstone-aware compaction), LevelDB has no way to prioritize reclaiming |
| 94 | > space from deleted keys. Restarting the daemon may trigger some compaction, |
| 95 | > but this is not guaranteed. |
| 96 | > |
| 97 | > If slow compaction is a problem, consider using the `pebbleds` datastore |
| 98 | > instead (see below), which handles this workload more efficiently. |
| 99 | |
| 100 | ## pebbleds |
| 101 | |
| 102 | Uses [pebble](https://github.com/cockroachdb/pebble) as a key-value store. |
| 103 | |
| 104 | ```json |
| 105 | { |
| 106 | "type": "pebbleds", |
| 107 | "path": "<location of pebble inside repo>", |
| 108 | } |
| 109 | ``` |
| 110 | |
| 111 | The following options are available for tuning pebble. |
| 112 | If they are not configured (or assigned their zero-valued), then default values are used. |
| 113 | |
| 114 | * `bytesPerSync`: int, Sync sstables periodically in order to smooth out writes to disk. (default: 512KB) |
| 115 | * `disableWAL`: true|false, Disable the write-ahead log (WAL) at expense of prohibiting crash recovery. (default: false) |
| 116 | * `cacheSize`: Size of pebble's shared block cache. (default: 8MB) |
| 117 | * `formatVersionMajor`: int, Sets the format of pebble on-disk files. If 0 or unset, automatically convert to latest format. |
| 118 | * `l0CompactionThreshold`: int, Count of L0 files necessary to trigger an L0 compaction. |
| 119 | * `l0StopWritesThreshold`: int, Limit on L0 read-amplification, computed as the number of L0 sublevels. |
| 120 | * `lBaseMaxBytes`: int, Maximum number of bytes for LBase. The base level is the level which L0 is compacted into. |
| 121 | * `maxConcurrentCompactions`: int, Maximum number of concurrent compactions. (default: 1) |
| 122 | * `memTableSize`: int, Size of a MemTable in steady state. The actual MemTable size starts at min(256KB, MemTableSize) and doubles for each subsequent MemTable up to MemTableSize (default: 4MB) |
| 123 | * `memTableStopWritesThreshold`: int, Limit on the number of queued of MemTables. (default: 2) |
| 124 | * `walBytesPerSync`: int: Sets the number of bytes to write to a WAL before calling Sync on it in the background. (default: 0, no background syncing) |
| 125 | * `walMinSyncSeconds`: int: Sets the minimum duration between syncs of the WAL. (default: 0) |
| 126 | |
| 127 | > [!TIP] |
| 128 | > Start using pebble with only default values and configure tuning items are needed for your needs. For a more complete description of these values, see: `https://pkg.go.dev/github.com/cockroachdb/pebble@vA.B.C#Options` (where `A.B.C` is pebble version from Kubo's `go.mod`). |
| 129 | |
| 130 | Using a pebble datastore can be set when initializing kubo `ipfs init --profile pebbleds`. |
| 131 | |
| 132 | #### Use of `formatMajorVersion` |
| 133 | |
| 134 | [Pebble's `FormatMajorVersion`](https://github.com/cockroachdb/pebble/tree/master?tab=readme-ov-file#format-major-versions) is a constant controlling the format of persisted data. Backwards incompatible changes to durable formats are gated behind new format major versions. |
| 135 | |
| 136 | At any point, a database's format major version may be bumped. However, once a database's format major version is increased, previous versions of Pebble will refuse to open the database. |
| 137 | |
| 138 | When IPFS is initialized to use the pebbleds datastore (`ipfs init --profile=pebbleds`), the latest pebble database format is configured in the pebble datastore config as `"formatMajorVersion"`. Setting this in the datastore config prevents automatically upgrading to the latest available version when kubo is upgraded. If a later version becomes available, the kubo daemon prints a startup message to indicate this. The user can them update the config to use the latest format when they are certain a downgrade will not be necessary. |
| 139 | |
| 140 | Without the `"formatMajorVersion"` in the pebble datastore config, the database format is automatically upgraded to the latest version. If this happens, then it is possible a downgrade back to the previous version of kubo will not work if new format is not compatible with the pebble datastore in the previous version of kubo. |
| 141 | |
| 142 | When installing a new version of kubo when `"formatMajorVersion"` is configured, migration does not upgrade this to the latest available version. This is done because a user may have reasons not to upgrade the pebble database format, and may want to be able to downgrade kubo if something else is not working in the new version. If the configured pebble database format in the old kubo is not supported in the new kubo, then the configured version must be updated and the old kubo run, before installing the new kubo. |
| 143 | |
| 144 | ## badgerds |
| 145 | |
| 146 | Uses [badger](https://github.com/dgraph-io/badger) as a key-value store. |
| 147 | |
| 148 | > [!CAUTION] |
| 149 | > **Badger v1 datastore is deprecated and will be removed in a future Kubo release.** |
| 150 | > |
| 151 | > This is based on very old badger 1.x, which has not been maintained by its |
| 152 | > upstream maintainers for years and has known bugs (startup timeouts, shutdown |
| 153 | > hangs, file descriptor |
| 154 | > exhaustion, and more). Do not use it for new deployments. |
| 155 | > |
| 156 | > **To migrate:** create a new `IPFS_PATH` with `flatfs` |
| 157 | > (`ipfs init --profile=flatfs`), move pinned data via |
| 158 | > `ipfs dag export/import` or `ipfs pin ls -t recursive|add`, and decommission the |
| 159 | > old badger-based node. When it comes to block storage, use experimental |
| 160 | > `pebbleds` only if you are sure modern `flatfs` does not serve your use case |
| 161 | > (most users will be perfectly fine with `flatfs`, it is also possible to keep |
| 162 | > `flatfs` for blocks and replace `leveldb` with `pebble` if preferred over |
| 163 | > `leveldb`). |
| 164 | |
| 165 | - `syncWrites`: Flush every write to disk before continuing. Setting this to false is safe as kubo will automatically flush writes to disk before and after performing critical operations like pinning. However, you can set this to true to be extra-safe (at the cost of a 2-3x slowdown when adding files). |
| 166 | - `truncate`: Truncate the DB if a partially written sector is found (defaults to true). There is no good reason to set this to false unless you want to manually recover partially written (and unpinned) blocks if kubo crashes half-way through a write operation. |
| 167 | |
| 168 | ```json |
| 169 | { |
| 170 | "type": "badgerds", |
| 171 | "path": "<location of badger inside repo>", |
| 172 | "syncWrites": true|false, |
| 173 | "truncate": true|false, |
| 174 | } |
| 175 | ``` |
| 176 | |
| 177 | ## mount |
| 178 | |
| 179 | Allows specified datastores to handle keys prefixed with a given path. |
| 180 | The mountpoints are added as keys within the child datastore definitions. |
| 181 | |
| 182 | ```json |
| 183 | { |
| 184 | "type": "mount", |
| 185 | "mounts": [ |
| 186 | { |
| 187 | // Insert other datastore definition here, but add the following key: |
| 188 | "mountpoint": "/path/to/handle" |
| 189 | }, |
| 190 | { |
| 191 | // Insert other datastore definition here, but add the following key: |
| 192 | "mountpoint": "/path/to/handle" |
| 193 | }, |
| 194 | ] |
| 195 | } |
| 196 | ``` |
| 197 | |
| 198 | ## measure |
| 199 | |
| 200 | This datastore is a wrapper that adds metrics tracking to any datastore. |
| 201 | |
| 202 | ```json |
| 203 | { |
| 204 | "type": "measure", |
| 205 | "prefix": "sometag.datastore", |
| 206 | "child": { datastore being wrapped } |
| 207 | } |
| 208 | ``` |
| 209 |