@cryptotaxi247 / kubo / commits / afab7659a

docs: clarify blockstore cache sizing and flatfs sharding (#11303)

* docs(config): clarify BlockKeyCacheSize and BloomFilterSize BlockKeyCacheSize was documented as "size in bytes" but the underlying boxo blockstore wires it directly to lru.New2Q[K,V](size int) which is an entry count, not a byte budget. Fix the unit and add memory sizing guidance (~200 B/entry) plus what the cache actually short-circuits (per-block flatfs Stat on the bitswap server hot path). BloomFilterSize section expanded with: what the filter answers (negative Has only), saturation behavior at runtime growth, startup AllKeysChan rebuild cost (one-time, scales with keyset not data volume), and a cross-link to BlockKeyCacheSize as the complementary positive-path cache. Drop the dead go-ipfs-blockstore link. * docs(datastores): explain flatfs next-to-last/3 for large blockstores The default next-to-last/2 shard depth (~1024 dirs) becomes a per-shard file-count problem on nodes growing past a few million blocks: bulk enumeration (GC, BloomFilterSize rebuild on startup, Provide.Strategy=all reprovider) and per-block Stat both pay readdir cost proportional to files-per-shard. next-to-last/3 (~32k dirs) keeps per-directory counts in a range modern filesystems handle well and is the recommended choice for pinning clusters, public gateways, and mirrors. Note that shard depth is fixed at ipfs init time and re-sharding requires a full export/import. * docs(config): expand BloomFilterSize sizing with bbloom specifics Replace the generic worked example with a power-of-two reference table covering 10M to 500M blocks, and document two kubo-specific behaviors that the generic bloom-filter math does not capture: - ipfs/bbloom rounds the bit count up to the next power of two, so non-power-of-two BloomFilterSize values silently allocate more memory than configured (e.g. the historical 1199120-byte example actually allocates a 2 MiB internal filter). - kubo wires bbloom with k=7 hash positions; the FPR formula is fixed at (1 - exp(-7n/m))^7. Memory cost is roughly ~1.2 B/entry at ~1% FPR and scales linearly with target FPR. Add a saturation section showing FPR degradation at 2x / 4x / 8x the design n (~11% / ~58% / >95% respectively), and a Risks subsection clarifying that a poorly sized filter is an operational waste rather than a correctness issue (no false negatives), with a quick bytes-per-block health check. Update the hur.st calculator URL from the n=1e6 default (dev-laptop scale) to n=10e6 (representative of real kubo deployments). Reference sizes verified empirically against ipfs/bbloom v0.1.0: a 16 MiB filter at n=10M gave 0.1875% observed FPR vs 0.18% predicted; the historical ~1.14 MiB worked example at n=1M gave 0.0545% vs 0.054% predicted at the rounded 2 MiB allocation. * docs(config): define FPR up front in BloomFilterSize section The BloomFilterSize section uses "FPR" throughout without defining it. Explain in the intro that the false-positive rate is the probability of a "maybe present" answer for a CID that is not actually in the blockstore, that a false positive costs at most one wasted datastore lookup (no data loss or incorrect retrieval), and that lower FPR means more inbound Has() calls answered from RAM alone. * docs: fold rounding penalty into bloom filter budget byte/entry sizing figures now report the operationally-useful number after bbloom's power-of-two rounding, so an operator following the guidance lands close to the true memory footprint instead of the raw design-point size. - config.md (BloomFilterSize): bump byte/entry to ~1.8/2.8/4.2 at ~1% / 0.1% / 0.01% FPR, state the average ~1.5x rounding penalty (worst case ~2x); drop 500M / 1 GiB row whose m/n=17.18 broke the uniform 10.74 ratio of the rest of the table; active-voice and comma fixes in saturation, risks, and startup prose - config.md (BlockKeyCacheSize): split a comma splice; active voice in 2Q replacement description - datastores.md (flatfs): align shard table columns; soften reshard wording to note kubo ships no in-place tool, not that flatfs forbids it

Marcin Rataj committed Apr 30, 2026 at 00:08 UTC afab7659aa36f05a8c6b06d91fda4188b0900d4c
2 files changed +171 -27
docs/config.md
+139 -26
@@ -966,25 +966,116 @@ Type: `bool`
966
967 ### `Datastore.BloomFilterSize`
968
969 -A number representing the size in bytes of the blockstore's [bloom
970 -filter](https://en.wikipedia.org/wiki/Bloom_filter). A value of zero represents
971 -the feature is disabled.
972 -
973 -This site generates useful graphs for various bloom filter values:
974 -<https://hur.st/bloomfilter/?n=1e6&p=0.01&m=&k=7> You may use it to find a
975 -preferred optimal value, where `m` is `BloomFilterSize` in bits. Remember to
976 -convert the value `m` from bits, into bytes for use as `BloomFilterSize` in the
977 -config file. For example, for 1,000,000 blocks, expecting a 1% false-positive
978 -rate, you'd end up with a filter size of 9592955 bits, so for `BloomFilterSize`
979 -we'd want to use 1199120 bytes. As of writing, [7 hash
980 -functions](https://github.com/ipfs/go-ipfs-blockstore/blob/547442836ade055cc114b562a3cc193d4e57c884/caching.go#L22)
981 -are used, so the constant `k` is 7 in the formula.
982 -
983 -Enabling the BloomFilter can provide performance improvements specially when
984 -responding to many requests for inexistent blocks. It however requires a full
985 -sweep of all the datastore keys on daemon start. On very large datastores this
986 -can be a very taxing operation, particularly if the datastore does not support
987 -querying existing keys without reading their values at the same time (blocks).
969 +The size in **bytes** of the blockstore's [bloom filter](https://en.wikipedia.org/wiki/Bloom_filter).
970 +A value of `0` disables the feature.
971 +
972 +The bloom filter answers "does the blockstore *not* have this CID?" from RAM
973 +without touching the datastore. A negative answer is exact (no false
974 +negatives, so blocks are never falsely reported missing); a positive answer
975 +is probabilistic and falls through to the underlying blockstore for
976 +verification. The chance of a false "maybe present" is the filter's
977 +**false-positive rate (FPR)**. A false positive costs one wasted datastore
978 +lookup; it never causes data loss or incorrect retrieval. The lower the FPR,
979 +the more `Has()` calls the filter answers from RAM alone.
980 +
981 +This cache pays off most on nodes that field many requests for content they
982 +don't host: public gateways, mirrors, and peers asked to serve
983 +opportunistically-cached blocks.
984 +
985 +The complementary cache for the *positive* path (block exists, look up its
986 +size) is [`Datastore.BlockKeyCacheSize`](#datastoreblockkeycachesize).
987 +
988 +#### How kubo's bloom filter is sized
989 +
990 +Kubo wires the underlying [`ipfs/bbloom`](https://github.com/ipfs/bbloom)
991 +filter with `k=7` hash positions. Two kubo-specific behaviors matter for
992 +sizing:
993 +
994 +1. **Power-of-two bit-count rounding.** bbloom rounds the requested bit
995 + count up to the next power of two, so a `BloomFilterSize` value that is
996 + not itself a power of two in bits silently allocates more memory than
997 + configured. For example, `BloomFilterSize: 1199120` (~1.14 MiB)
998 + actually allocates a `16,777,216`-bit (= 2 MiB) filter internally. For
999 + predictable behavior, pick `BloomFilterSize` values that are
1000 + power-of-two byte counts: 1 MiB, 2 MiB, 4 MiB, ..., 256 MiB, 512 MiB,
1001 + 1 GiB.
1002 +2. **Fixed `k=7`.** With seven hash positions, FPR for a filter of `m`
1003 + bits and `n` inserted entries is `(1 - exp(-7n/m))^7`. To hit a
1004 + target FPR, budget roughly ~1.8 bytes per entry at ~1% FPR, ~2.8
1005 + bytes per entry at ~0.1% FPR, and ~4.2 bytes per entry at ~0.01%
1006 + FPR. These figures already include the average ~1.5x penalty from
1007 + the power-of-two rounding above; the worst case is ~2x.
1008 +
1009 +#### Reference sizing
1010 +
1011 +Power-of-two `BloomFilterSize` values for common blockset sizes, with the
1012 +FPR you can expect at the design point and at 2× growth:
1013 +
1014 +| Expected blocks (`n`) | `BloomFilterSize` | FPR at `n` | FPR at 2× `n` |
1015 +|---:|---:|---:|---:|
1016 +| 10,000,000 | `16777216` (16 MiB) | ~0.18% | ~5% |
1017 +| 25,000,000 | `33554432` (32 MiB) | ~0.58% | ~11% |
1018 +| 50,000,000 | `67108864` (64 MiB) | ~0.58% | ~11% |
1019 +| 100,000,000 | `134217728` (128 MiB) | ~0.58% | ~11% |
1020 +| 200,000,000 | `268435456` (256 MiB) | ~0.58% | ~11% |
1021 +
1022 +For a tighter FPR at the design point, step up to the next power of two.
1023 +
1024 +The [hur.st/bloomfilter](https://hur.st/bloomfilter/?n=10e6&p=0.01&m=&k=7)
1025 +calculator works as a reference for exploring `(n, p, m)` combinations
1026 +(remember kubo uses `k=7`); just keep in mind that the `m` it suggests
1027 +is the optimal-fit value, while bbloom rounds up to the next power of
1028 +two on top of that.
1029 +
1030 +#### Saturation as the repo grows
1031 +
1032 +A bloom filter is fixed-size after creation. As more CIDs are inserted
1033 +past its design `n`, the false-positive rate climbs steeply. Rough
1034 +behavior with a filter sized for ~0.6% FPR at its design point:
1035 +
1036 +- At `n`: ~0.6% FPR. Every "definitely not" reliably saves a datastore
1037 + lookup.
1038 +- At ~`2 × n`: ~11% FPR. Most negatives still save lookups, but tail
1039 + latency rises because each "maybe" still hits the datastore.
1040 +- At ~`4 × n`: ~58% FPR. Most "maybe" answers fall through. The filter
1041 + is mostly paying CPU and RAM cost without short-circuiting much.
1042 +- At ~`8 × n` or more: above ~95% FPR. Effectively saturated. The
1043 + filter answers "maybe" for nearly every CID and provides no benefit.
1044 +
1045 +Size for **expected steady-state, not today's count**, and re-tune after
1046 +crossing the design point. Bloom filters cannot grow in place; raising
1047 +`BloomFilterSize` and restarting the daemon rebuilds the filter from
1048 +scratch.
1049 +
1050 +#### Risks of an undersized filter
1051 +
1052 +A poorly-sized filter is **never a correctness issue**. Bloom filters
1053 +have no false negatives, so blocks are never falsely reported missing.
1054 +The risks are operational:
1055 +
1056 +- **Wasted RAM and CPU.** Every `Has()` still runs all seven hash
1057 + positions. Once the filter saturates, those cycles return nothing.
1058 +- **Silent regression as the pinset grows.** A filter sized for last
1059 + year's data can drift past saturation without warning; the
1060 + negative-`Has` short-circuit benefit just quietly disappears.
1061 +- **Recurring startup tax.** The filter rebuilds on every daemon
1062 + restart (see below). On slow disks this means minutes of
1063 + `AllKeysChan` walking, paid in full even when the resulting filter
1064 + is too small to help.
1065 +
1066 +Quick health check: divide `BloomFilterSize` by your current block count.
1067 +Below ~`1` byte/block the filter is past its design point; below
1068 +~`0.5` bytes/block it is effectively saturated.
1069 +
1070 +#### Startup cost
1071 +
1072 +The filter is not persisted across restarts. Every daemon start rebuilds it
1073 +by walking all datastore keys (`AllKeysChan`). On very large blockstores or
1074 +slow disks this can take many minutes, during which `Has()` falls through
1075 +to the datastore and the filter provides no benefit. Datastores that cannot
1076 +enumerate keys without reading values (block content) pay even more here;
1077 +flatfs and pebble both support keys-only iteration, so the rebuild cost
1078 +scales with the keyset, not data volume.
1079
1080 Default: `0` (disabled)
1081
@@ -1011,16 +1102,38 @@ Type: `bool`
1102
1103 ### `Datastore.BlockKeyCacheSize`
1104
1014 -A number representing the maximum size in bytes of the blockstore's Two-Queue
1015 -cache, which caches block-cids and their block-sizes. Use `0` to disable.
1105 +The maximum **number of entries** held in the blockstore's Two-Queue cache. The
1106 +cache stores per-CID metadata (existence and block size) but never block
1107 +content. Use `0` to disable.
1108
1017 -This cache, once primed, can greatly speed up operations like `ipfs repo stat`
1018 -as there is no need to read full blocks to know their sizes. Size should be
1019 -adjusted depending on the number of CIDs on disk (`NumObjects in`ipfs repo stat`).
1109 +A cache hit answers `Has` and `GetSize` from RAM and skips the underlying
1110 +datastore lookup. This includes the per-block `os.Stat` flatfs does to learn a
1111 +block's size, which is the dominant cost on bitswap servers responding to peer
1112 +wantlists.
1113
1021 -Default: `65536` (64KiB)
1114 +The cache uses a [Two-Queue (2Q) replacement policy](https://pkg.go.dev/github.com/hashicorp/golang-lru/v2#TwoQueueCache):
1115 +an entry must be touched twice before it is promoted to the frequently-used
1116 +tier. A long one-shot scan (reprovider, GC, `ipfs repo verify`) therefore
1117 +does not evict the hot entries that bitswap repeatedly serves.
1118
1023 -Type: `optionalInteger` (non-negative, bytes)
1119 +#### Sizing
1120 +
1121 +Memory usage is roughly the entry count times the per-entry overhead, which
1122 +combines 2Q bookkeeping, the multihash key bytes, and the cached value. As a
1123 +rough estimate, budget ~200 bytes per entry, so `1048576` (1M entries) is on
1124 +the order of ~200 MB resident. The cache only needs to cover the **hot
1125 +working set** of CIDs (the ones repeatedly hit by inbound bitswap, gateway,
1126 +or DAG-resolution traffic), not the entire blockstore.
1127 +
1128 +The default of `65536` is sized for small dev/desktop nodes. Operators
1129 +running public gateways, pinning clusters, or any node serving non-trivial
1130 +bitswap traffic should size this against the active working set. See
1131 +[`Datastore.BloomFilterSize`](#datastorebloomfiltersize) for the
1132 +complementary negative-`Has()` short-circuit that pairs well with this cache.
1133 +
1134 +Default: `65536` (entries)
1135 +
1136 +Type: `optionalInteger` (non-negative, number of entries)
1137
1138 ### `Datastore.Spec`
1139
docs/datastores.md
+32 -1
@@ -16,7 +16,9 @@ Stores each key-value pair as a file on the filesystem.
16
17 The shardFunc is prefixed with `/repo/flatfs/shard/v1` then followed by a descriptor of the sharding strategy. Some example values are:
18 - `/repo/flatfs/shard/v1/next-to-last/2`
19 - - Shards on the two next to last characters of the key
19 + - Shards on the two next-to-last base32 characters of the key (~1024 directories)
20 +- `/repo/flatfs/shard/v1/next-to-last/3`
21 + - Shards on the three next-to-last base32 characters of the key (~32,768 directories)
22 - `/repo/flatfs/shard/v1/prefix/2`
23 - Shards based on the two-character prefix of the key
24
@@ -33,6 +35,35 @@ The shardFunc is prefixed with `/repo/flatfs/shard/v1` then followed by a descri
35
36 NOTE: flatfs must only be used as a block store (mounted at `/blocks`) as it only partially implements the datastore interface. You can mount flatfs for /blocks only using the mount datastore (described below).
37
38 +### Choosing a `shardFunc` for large blockstores
39 +
40 +The `next-to-last/N` shard depth controls how many directories the blockstore
41 +is spread across. Each shard becomes a single directory under `blocks/`, and
42 +every block file lives directly inside its shard. The cost of any operation
43 +that does a `readdir` or per-file `stat` on a shard scales with the number of
44 +files in that shard.
45 +
46 +Two depths in common use:
47 +
48 +| `shardFunc` | Shard count | At 60M blocks | Notes |
49 +|--------------------------|------------:|----------------:|---------------------------------------------|
50 +| `next-to-last/2` | ~1,024 | ~58k files/dir | default; fine for small/medium nodes |
51 +| `next-to-last/3` | ~32,768 | ~1.8k files/dir | recommended for large pinning/gateway nodes |
52 +
53 +For nodes expected to grow past a few million blocks (most pinning clusters,
54 +public gateways, mirrors), prefer `next-to-last/3`. The deeper sharding keeps
55 +per-directory file counts in a range modern filesystems handle well, and it
56 +significantly reduces the per-operation cost of `Stat`, `readdir`, and bulk
57 +enumeration (used by GC, [`Datastore.BloomFilterSize`](config.md#datastorebloomfiltersize)
58 +rebuild on startup, and `Provide.Strategy=all` reprovide cycles). On nodes
59 +backed by rotational disks the difference can be the gap between healthy
60 +operation and IOPS-saturated iowait.
61 +
62 +The shard depth is fixed at `ipfs init` time. Kubo ships no in-place
63 +re-sharding tool, so switching depth on an existing repo means exporting
64 +and re-importing the blockstore. Pick conservatively for the expected
65 +steady state of the node.
66 +
67 ## levelds
68
69 Uses a [leveldb](https://github.com/syndtr/goleveldb) database to store key-value