@cryptotaxi247 / kubo / commits / 356d26149

feat(provide): +unique and +entities strategy modifiers (#11245)

* fix(config): harden provide strategy parsing with error returns - config: ParseProvideStrategy returns error, rejects "all" mixed with selective strategies, removes dead strategy==0 check - config: add MustParseProvideStrategy for pre-validated call sites - config: ValidateProvideConfig validates strategy at startup - config: ShouldProvideForStrategy uses bitmask check for ProvideStrategyAll - core/node: downstream callers use MustParseProvideStrategy - core/node: fix Pinning() nil return that caused fx.Provide panic * feat(config): add +unique and +entities strategy modifiers - ProvideStrategyUnique: bloom filter cross-DAG deduplication - ProvideStrategyEntities: entity-aware traversal (implies Unique) - parser: "unique" and "entities" tokens recognized - validation: modifiers must combine with pinned/mfs, incompatible with all/roots - go.mod: update boxo to feat/provide-entity-roots-with-dedup (VisitedTracker, WalkDAG, WalkEntityRoots, NewConcatProvider, NewUniquePinnedProvider, NewPinnedEntityRootsProvider) * refactor(cmd): rename ExecuteFastProvide to ExecuteFastProvideRoot pure rename, no behavior change. prepares for ExecuteFastProvideDAG which will walk the DAG according to Provide.Strategy. * feat(pin): fast-provide root CID after pin add and pin update adds ExecuteFastProvideRoot calls to pin add and pin update, matching the behavior of ipfs add and ipfs dag import. respects Import.FastProvideRoot and Import.FastProvideWait config options. previously, pin add/update did not trigger any immediate providing, leaving pinned content invisible to the DHT until the next reprovide cycle (up to 22h). * feat(provider): wire +unique reprovide cycle with bloom dedup when Provide.Strategy includes +unique, the reprovide cycle uses a shared BloomTracker across all sub-walks (MFS, recursive pins, direct pins). duplicate sub-DAG branches across recursive pins are detected and skipped, reducing traversal from O(pins * total_blocks) to O(unique_blocks). - readLastUniqueCount / persistUniqueCount: persist bloom sizing count between cycles at /reprovideLastUniqueCount - uniqueMFSProvider: MFS walker with shared tracker + locality check - createKeyProvider restructured: +unique bit checked first, non-unique strategies fall through to existing switch unchanged - per-cycle fresh BloomTracker sized from previous cycle's count - channel wrapper persists count on successful cycle completion * feat(provider): wire +entities reprovide cycle with entity root walkers when Provide.Strategy includes +entities (which implies +unique), the reprovide cycle uses WalkEntityRoots instead of WalkDAG, emitting only entity roots (files, directories, HAMT shards) and skipping internal file chunks. - mfsEntityRootsProvider: MFS walk with entity root detection - createKeyProvider: select walker based on +entities flag via function references (makePinProv / makeMFSProv) to avoid duplicating the stream wiring logic - all combinations: pinned+entities, mfs+entities, pinned+mfs+entities * docs: document +unique and +entities strategy modifiers - config.md: document +unique, +entities modifiers with caveats (range request limitation, roots vs entities distinction) - changelog v0.41: add entries for strategy modifiers, pin add/update fast-provide, and hardened strategy parsing * feat: gate providingDagService behind --fast-provide-dag per-block providing during ipfs add is now opt-in via --fast-provide-dag (or Import.FastProvideDAG config, default: false). without it, only the root CID is fast-provided after add, and the reprovide cycle handles the rest. this changes the default for Provide.Strategy=pinned: previously every block was provided during write, now only the root is immediate. use --fast-provide-dag=true to restore the previous behavior. Provide.Strategy=all is unaffected (blockstore hook provides on Put). * feat(pin): expose --fast-provide-root and --fast-provide-wait flags pin add and pin update now accept the same --fast-provide-root and --fast-provide-wait CLI flags as ipfs add and ipfs dag import, with the same config fallbacks (Import.FastProvideRoot, Import.FastProvideWait). previously these were config-only with no CLI override. * feat: wire --fast-provide-dag across all content commands --fast-provide-dag now available on ipfs add, ipfs dag import, ipfs pin add, and ipfs pin update (matching --fast-provide-root). - ExecuteFastProvideDAG accepts []cid.Cid so multiple roots share one bloom tracker (cross-root dedup for dag import and pin add) - --fast-provide-dag supersedes --fast-provide-root (DAG walk includes the root CID as the first emitted via DFS pre-order) - wait parameter: when true blocks until walk completes, when false runs in background goroutine - Import.FastProvideDAG config option (default: false) * docs(config): improve Provide.Strategy docs, add Import.FastProvideDAG - strategy section: clearer trade-offs, suggested configurations, memory comparison with concrete numbers - Import.FastProvideDAG: new config option documentation - Import.FastProvideRoot/Wait: updated to mention pin commands - all three Import.FastProvide* options: consistent "Applies to" lists * chore: gofumpt and gci formatting * chore: update boxo to latest feat/provide-entity-roots-with-dedup * feat: TEST_DHT_STUB with ephemeral DHT peers when TEST_DHT_STUB=1, the CLI test harness creates 20 in-process libp2p hosts on loopback, each running a DHT server with a shared in-memory ProviderStore. kubo daemons bootstrap to them over real TCP, exercising the full DHT code path without public internet. tests opt in via h.SetStubBootstrap(nodes) after Init(). on the daemon side, WAN DHT filters (AddressFilter, QueryFilter, RoutingTableFilter, RoutingTablePeerDiversityFilter) are lifted to accept loopback peers when TEST_DHT_STUB is set. depends on: github.com/libp2p/go-libp2p-kad-dht#1241 * test: harden provider strategy tests add sweep reprovide tests for all strategies (all, pinned, roots, mfs, pinned+mfs). each test waits for two reprovide cycles to confirm the schedule runs repeatedly. sweep uses short Provide.DHT.Interval and polls provide stat --enc=json. harden negative assertions: - roots: test excludes child blocks of a recursive pin (not just unpinned content), using --only-hash to learn the child CID - mfs: test that pinned content outside MFS is not provided fix: ipfs add --only-hash no longer triggers fast-provide or pinning (was providing CIDs for data that was never stored) rename SetStubBootstrap to BootstrapWithStubDHT with lazy-init (ephemeral peers created on first call, not on harness creation) * test: add +unique and +entities strategy tests strategy tests for pinned+mfs+unique and pinned+mfs+entities, covering both provide-at-add-time and reprovide (two cycles). content uses a nested DAG (root/subdir/largefile with 1 MiB chunks) to exercise the walker on multi-level structures. BootstrapWithStubDHT is now self-contained: it always creates 20 ephemeral DHT peers on loopback and sets TEST_DHT_STUB=1 on each node's environment so the daemon lifts WAN DHT filters. no external env var needed. the sweep provider requires >=20 DHT peers to estimate network size (prefix length); without enough peers it stays offline and never provides. TEST_DHT_STUB on the daemon side lifts WAN DHT filters (AddressFilter, QueryFilter, RoutingTableFilter, RoutingTablePeerDiversityFilter) to accept loopback peers. this is set automatically by BootstrapWithStubDHT. other changes: - Provide.DHT.Interval=30s in sweep reprovide tests (was 1m) - uniq() helper for unique CIDs across parallel subtests - ipfs add --only-hash disables fast-provide and pinning * docs: improve help text and changelog accuracy ipfs add --help: rewrite fast-provide section with clear structure (content discoverability, flag defaults, strategy=all behavior) ipfs routing reprovide: mark as deprecated, note it returns an error with sweep provider, log error with actionable guidance changelog: fix missing --fast-provide-dag flag on pin commands, use "routing system" instead of "DHT" where applicable, link to docs/config.md as source of truth for defaults environment-variables.md: note that BootstrapWithStubDHT sets TEST_DHT_STUB automatically, no external env var needed * chore: revert go-libp2p-kad-dht to released v0.39.0 the fork (NoopMessageSender, MsgSenderBuilder) is no longer used. the ephemeral peer pool in BootstrapWithStubDHT replaced the NoopMessageSender approach. * feat: log bloom dedup stats after provide cycles log providedCIDs and skippedBranches after each unique reprovide cycle and fast-provide-dag walk. tests verify exact counts with two dir pins sharing a 10 KiB file (5 KiB chunks): fast-provide-dag asserts 5 provided + 1 skipped branch, reprovide asserts 6 provided + 1 skipped branch (includes empty MFS root pin). both assert bloom tracker created and no autoscale. updates boxo to pick up Deduplicated() counter, bloom creation/autoscale logging, and review feedback fixes. * chore(deps): switch boxo to post-merge commit boxo#1124 landed on master; point to the merge commit instead of the PR branch. * fix(coreapi): drop providingDagService wrap ipfs add --pin --fast-provide-dag wrapped the DAGService with providingDagService, which announced every block as it was written regardless of strategy modifiers. ExecuteFastProvideDAG ran in parallel as the post-add walker. Net effect: - pinned+entities: chunks reached the DHT despite +entities saying they should be skipped (correctness bug) - pinned+unique: every block announced twice; the post-walk bloom only dedups against its own pass - pinned (plain): every block announced twice ExecuteFastProvideDAG already has bloom dedup, entity-roots support, and unbuffered backpressure, so it is now the single mechanism for --fast-provide-dag across ipfs add, dag import, pin add, and pin update. Provide.Strategy=all is untouched: every block is provided at the blockstore level via the blockstore.Provider hook in core/node/storage.go, which is independent of coreapi. The Pinned strategy bit gated providingDagService and the parser rejects combining "all" with other strategies, so "all" never set that bit in the first place. - core/coreapi/unixfs.go: drop the wrap, the providingDagService struct, and the now-unused mh and boxo/provider imports - core/coreiface/options/unixfs.go: drop FastProvideDAG option - core/coreapi/coreapi.go: drop now-dead providingStrategy field - core/commands/add.go: drop the FastProvideDAG option pass-through - test/cli/provider_test.go: regression test using ipfs add --fast-provide-dag with pinned+entities -- fails on the previous code and passes here * feat(config): add Provide.BloomFPRate Operators tuning +unique or +entities strategies on memory-constrained or extra-large repos previously had no way to trade bloom filter memory against false-positive rate -- both the reprovide cycle and fast-provide-dag walks hardcoded walker.DefaultBloomFPRate. Provide.BloomFPRate is the target false positive rate (1/N) for the shared bloom tracker. Has no effect on Provide.Strategy=all or other strategies that do not walk DAGs through the tracker. Validation rejects values below 1_000_000 (~1 in 1M); below that the bloom becomes lossy enough to drop a meaningful fraction of CIDs from each reprovide cycle. The single source of truth for the default value is config.DefaultProvideBloomFPRate; docs reference it descriptively (~1 in 4.75M, ~4 bytes/CID) so the literal lives in exactly one place. - config/provide.go: BloomFPRate field, DefaultProvideBloomFPRate and MinProvideBloomFPRate constants, validation - config/provide_test.go: round-trip + validation cases - core/node/provider.go: plumb fpRate through setReproviderKeyProvider and createKeyProvider - core/commands/cmdenv/env.go: ExecuteFastProvideDAG takes fpRate - core/commands/{add,dag/import,pin/pin}.go: resolve from cfg and pass through to ExecuteFastProvideDAG - docs/config.md: new Provide.BloomFPRate section after Provide.DHT.* with memory tradeoff table and minimum-value note - docs/changelogs/v0.41.md: link to the new option from the +unique/ +entities section * test(node): cover unique count persistence readLastUniqueCount and persistUniqueCount were exercised only indirectly via CLI tests, leaving the 8-byte length check and the "missing key" fallback without direct coverage. - empty datastore returns 0 (no previous cycle) - round trip across the full uint64 range (0, 1, 1k, 1M, 1B, MaxUint64) - overwrite returns the most recent value (matches per-cycle persist) - corrupt length (empty, short, long, single byte) returns 0 instead of panicking * fix(cmdenv): tie async fast-provide to node ctx Background fast-provide goroutines were implicitly bound to req.Context, which go-ipfs-cmds cancels on handler exit, so async --fast-provide-dag (and --fast-provide-root parented on context.Background) aborted or outlived the node. Parent both paths off the IpfsNode lifetime context instead. - ExecuteFastProvideRoot: async goroutine now derives from ipfsNode.Context(), so it cancels on daemon shutdown rather than potentially touching a closed DHT client. - ExecuteFastProvideDAG: takes cmdCtx and nodeCtx; wait=true runs inline under cmdCtx (Ctrl+C still cancels the walk), wait=false runs in a goroutine under nodeCtx so the walk survives command exit but still stops on shutdown. - add, dag import, pin add/update: pass node.Context() as the new nodeCtx argument. - changelog: note the behavior change for opt-in strategies. * test(cli): cover async fast-provide-dag walk Adds TestProviderFastProvideDAGAsyncSurvives: ipfs add with --fast-provide-dag=true but no --fast-provide-wait must walk the full DAG in a background goroutine that outlives the command handler, announce every block, and leave chunk CIDs findable by peers via findprovs. A long Provide.DHT.Interval ensures the scheduled reprovide cycle cannot be the source of the chunk announcements. * docs(changelog): tighten v0.41 provide section

Marcin Rataj committed Apr 10, 2026 at 02:53 UTC 356d261490a9fbabbdbe624ccd752c91a8369126

27 files changed +1925 -310

config/import.go

@@ -23,6 +23,7 @@ const (

23 DefaultHashFunction = "sha2-256"

24 DefaultFastProvideRoot = true

25 DefaultFastProvideWait = false

26 + DefaultFastProvideDAG = false

28 DefaultUnixFSHAMTDirectorySizeThreshold = 262144 // 256KiB - https://github.com/ipfs/boxo/blob/6c5a07602aed248acc86598f30ab61923a54a83e/ipld/unixfs/io/directory.go#L26

@@ -71,6 +72,7 @@ type Import struct {

72 BatchMaxNodes OptionalInteger

73 BatchMaxSize OptionalInteger

74 FastProvideRoot Flag

75 + FastProvideDAG Flag

76 FastProvideWait Flag

77 }

config/provide.go

+85 -4

@@ -12,6 +12,22 @@ const (

12 DefaultProvideEnabled = true

13 DefaultProvideStrategy = "all"

15 + // DefaultProvideBloomFPRate is the target false positive rate for the

16 + // bloom filter used by +unique and +entities reprovide cycles and

17 + // fast-provide-dag walks. Expressed as 1/N (one false positive per N

18 + // lookups). At ~1 in 4.75M (~0.00002%) each CID costs ~4 bytes before

19 + // ipfs/bbloom's power-of-two rounding.

20 + //

21 + // Kubo owns this default independently of boxo/dag/walker; the two

22 + // values may diverge over time without coordination.

23 + DefaultProvideBloomFPRate = 4_750_000

24 +

25 + // MinProvideBloomFPRate is the smallest accepted Provide.BloomFPRate.

26 + // Below 1 in 1M the bloom filter becomes lossy enough to drop a

27 + // meaningful fraction of CIDs from each reprovide cycle (e.g. at

28 + // rate=10_000 a 100M-CID repo skips ~10K CIDs per cycle).

29 + MinProvideBloomFPRate = 1_000_000

30 +

31 // DHT provider defaults

32 DefaultProvideDHTInterval = 22 * time.Hour // https://github.com/ipfs/kubo/pull/9326

33 DefaultProvideDHTMaxWorkers = 16 // Unified default for both sweep and legacy providers

@@ -36,6 +52,8 @@ const (

52 ProvideStrategyPinned

53 ProvideStrategyRoots

54 ProvideStrategyMFS

55 + ProvideStrategyUnique // bloom filter cross-DAG deduplication

56 + ProvideStrategyEntities // entity-aware traversal (implies Unique)

57 )

59 // Provide configures both immediate CID announcements (provide operations) for new content

@@ -50,6 +68,16 @@ type Provide struct {

68 // Default: DefaultProvideStrategy

69 Strategy *OptionalString `json:",omitempty"`

71 + // BloomFPRate sets the target false positive rate of the bloom filter

72 + // used by Provide.Strategy modifiers +unique and +entities (and the

73 + // matching fast-provide-dag walk). Expressed as 1/N (one false

74 + // positive per N lookups), so higher N means lower FP rate but more

75 + // memory per CID. Only takes effect when Provide.Strategy includes

76 + // +unique or +entities.

77 + //

78 + // Default: DefaultProvideBloomFPRate

79 + BloomFPRate *OptionalInteger `json:",omitempty"`

80 +

81 // DHT configures DHT-specific provide and reprovide settings.

82 DHT ProvideDHT

83 }

@@ -100,25 +128,78 @@ type ProvideDHT struct {

128 ResumeEnabled Flag `json:",omitempty"`

129 }

130

103 -func ParseProvideStrategy(s string) ProvideStrategy {

131 +func ParseProvideStrategy(s string) (ProvideStrategy, error) {

132 var strategy ProvideStrategy

133 for part := range strings.SplitSeq(s, "+") {

134 switch part {

107 - case "all", "flat", "": // special case, does not mix with others ("flat" is deprecated, maps to "all")

108 - return ProvideStrategyAll

135 + case "all", "flat":

136 + strategy |= ProvideStrategyAll

137 + case "":

138 + // empty string (default config) maps to "all",

139 + // but empty tokens from splitting (e.g. "pinned+") are invalid

140 + if s == "" {

141 + strategy |= ProvideStrategyAll

142 + } else {

143 + return 0, fmt.Errorf("invalid provide strategy: empty token in %q", s)

144 + }

145 case "pinned":

146 strategy |= ProvideStrategyPinned

147 case "roots":

148 strategy |= ProvideStrategyRoots

149 case "mfs":

150 strategy |= ProvideStrategyMFS

151 + case "unique":

152 + strategy |= ProvideStrategyUnique

153 + case "entities":

154 + strategy |= ProvideStrategyEntities | ProvideStrategyUnique

155 + default:

156 + return 0, fmt.Errorf("unknown provide strategy token: %q in %q", part, s)

157 + }

158 + }

159 + // "all" provides every block and cannot be combined with selective strategies

160 + if strategy&ProvideStrategyAll != 0 && strategy != ProvideStrategyAll {

161 + return 0, fmt.Errorf("\"all\" strategy cannot be combined with other strategies in %q", s)

162 + }

163 + // +unique/+entities require a base strategy that walks DAGs (pinned and/or mfs)

164 + wantsDedup := strategy&(ProvideStrategyUnique|ProvideStrategyEntities) != 0

165 + if wantsDedup {

166 + walksDAGs := strategy&(ProvideStrategyPinned|ProvideStrategyMFS) != 0

167 + if !walksDAGs {

168 + return 0, fmt.Errorf("+unique/+entities must combine with pinned and/or mfs in %q", s)

169 + }

170 + if strategy&ProvideStrategyRoots != 0 {

171 + return 0, fmt.Errorf("+unique/+entities is incompatible with roots in %q", s)

172 }

173 }

174 + return strategy, nil

175 +}

176 +

177 +// MustParseProvideStrategy is like ParseProvideStrategy but panics on error.

178 +// Use with strategy strings that have already been validated at startup.

179 +func MustParseProvideStrategy(s string) ProvideStrategy {

180 + strategy, err := ParseProvideStrategy(s)

181 + if err != nil {

182 + panic(err)

183 + }

184 return strategy

185 }

186

187 // ValidateProvideConfig validates the Provide configuration according to DHT requirements.

188 func ValidateProvideConfig(cfg *Provide) error {

189 + // Validate Provide.Strategy

190 + strategy := cfg.Strategy.WithDefault(DefaultProvideStrategy)

191 + if _, err := ParseProvideStrategy(strategy); err != nil {

192 + return fmt.Errorf("Provide.Strategy: %w", err)

193 + }

194 +

195 + // Validate Provide.BloomFPRate

196 + if !cfg.BloomFPRate.IsDefault() {

197 + rate := cfg.BloomFPRate.WithDefault(DefaultProvideBloomFPRate)

198 + if rate < MinProvideBloomFPRate {

199 + return fmt.Errorf("Provide.BloomFPRate must be >= %d (1 in 1M), got %d", MinProvideBloomFPRate, rate)

200 + }

201 + }

202 +

203 // Validate Provide.DHT.Interval

204 if !cfg.DHT.Interval.IsDefault() {

205 interval := cfg.DHT.Interval.WithDefault(DefaultProvideDHTInterval)

@@ -184,7 +265,7 @@ func ValidateProvideConfig(cfg *Provide) error {

265 // ShouldProvideForStrategy determines if content should be provided based on the provide strategy

266 // and content characteristics (pinned status, root status, MFS status).

267 func ShouldProvideForStrategy(strategy ProvideStrategy, isPinned bool, isPinnedRoot bool, isMFS bool) bool {

187 - if strategy == ProvideStrategyAll {

268 + if strategy&ProvideStrategyAll != 0 {

269 // 'all' strategy: always provide

270 return true

271 }

config/provide_test.go

+181 -19

@@ -9,27 +9,146 @@ import (

9 )

11 func TestParseProvideStrategy(t *testing.T) {

12 - tests := []struct {

13 - input string

14 - expect ProvideStrategy

15 - }{

16 - {"all", ProvideStrategyAll},

17 - {"pinned", ProvideStrategyPinned},

18 - {"mfs", ProvideStrategyMFS},

19 - {"pinned+mfs", ProvideStrategyPinned | ProvideStrategyMFS},

20 - {"invalid", 0},

21 - {"all+invalid", ProvideStrategyAll},

22 - {"", ProvideStrategyAll},

23 - {"flat", ProvideStrategyAll}, // deprecated, maps to "all"

24 - {"flat+all", ProvideStrategyAll},

25 - }

12 + t.Run("valid strategies", func(t *testing.T) {

13 + tests := []struct {

14 + input string

15 + expect ProvideStrategy

16 + }{

17 + {"all", ProvideStrategyAll},

18 + {"pinned", ProvideStrategyPinned},

19 + {"roots", ProvideStrategyRoots},

20 + {"mfs", ProvideStrategyMFS},

21 + {"pinned+mfs", ProvideStrategyPinned | ProvideStrategyMFS},

22 + {"pinned+roots", ProvideStrategyPinned | ProvideStrategyRoots},

23 + {"pinned+mfs+roots", ProvideStrategyPinned | ProvideStrategyMFS | ProvideStrategyRoots},

24 + {"", ProvideStrategyAll}, // empty string = default = all

25 + {"flat", ProvideStrategyAll}, // deprecated, maps to "all"

26 + {"flat+all", ProvideStrategyAll}, // redundant but valid

27 + {"all+all", ProvideStrategyAll}, // redundant but valid

28 + {"mfs+pinned", ProvideStrategyMFS | ProvideStrategyPinned}, // order doesn't matter

29 + // +unique and +entities modifiers

30 + {"pinned+unique", ProvideStrategyPinned | ProvideStrategyUnique},

31 + {"pinned+entities", ProvideStrategyPinned | ProvideStrategyEntities | ProvideStrategyUnique},

32 + {"pinned+unique+entities", ProvideStrategyPinned | ProvideStrategyUnique | ProvideStrategyEntities},

33 + {"mfs+unique", ProvideStrategyMFS | ProvideStrategyUnique},

34 + {"mfs+entities", ProvideStrategyMFS | ProvideStrategyEntities | ProvideStrategyUnique},

35 + {"pinned+mfs+unique", ProvideStrategyPinned | ProvideStrategyMFS | ProvideStrategyUnique},

36 + {"pinned+mfs+entities", ProvideStrategyPinned | ProvideStrategyMFS | ProvideStrategyEntities | ProvideStrategyUnique},

37 + }

27 - for _, tt := range tests {

28 - result := ParseProvideStrategy(tt.input)

29 - if result != tt.expect {

30 - t.Errorf("ParseProvideStrategy(%q) = %d, want %d", tt.input, result, tt.expect)

39 + for _, tt := range tests {

40 + result, err := ParseProvideStrategy(tt.input)

41 + require.NoError(t, err, "ParseProvideStrategy(%q)", tt.input)

42 + assert.Equal(t, tt.expect, result, "ParseProvideStrategy(%q)", tt.input)

43 }

32 - }

44 + })

45 +

46 + t.Run("unknown token (including typos)", func(t *testing.T) {

47 + tests := []struct {

48 + input string

49 + err string

50 + }{

51 + {"invalid", `unknown provide strategy token: "invalid"`},

52 + {"uniuqe", `unknown provide strategy token: "uniuqe"`}, // typo of "unique"

53 + {"entites", `unknown provide strategy token: "entites"`}, // cspell:disable-line -- intentional typo of "entities"

54 + {"pinned+uniuqe", `unknown provide strategy token: "uniuqe"`}, // typo in combo

55 + }

56 +

57 + for _, tt := range tests {

58 + _, err := ParseProvideStrategy(tt.input)

59 + require.Error(t, err, "ParseProvideStrategy(%q) should fail", tt.input)

60 + assert.Contains(t, err.Error(), tt.err)

61 + }

62 + })

63 +

64 + t.Run("empty token from delimiter", func(t *testing.T) {

65 + tests := []string{

66 + "pinned+", // trailing +

67 + "+pinned", // leading +

68 + "pinned++mfs", // double +

69 + }

70 +

71 + for _, input := range tests {

72 + _, err := ParseProvideStrategy(input)

73 + require.Error(t, err, "ParseProvideStrategy(%q) should fail", input)

74 + assert.Contains(t, err.Error(), "empty token")

75 + }

76 + })

77 +

78 + t.Run("all cannot be combined with other strategies", func(t *testing.T) {

79 + tests := []string{

80 + "all+pinned",

81 + "all+mfs",

82 + "all+roots",

83 + "flat+pinned",

84 + "all+pinned+mfs",

85 + }

86 +

87 + for _, input := range tests {

88 + _, err := ParseProvideStrategy(input)

89 + require.Error(t, err, "ParseProvideStrategy(%q) should fail", input)

90 + assert.Contains(t, err.Error(), "cannot be combined")

91 + }

92 + })

93 +

94 + t.Run("+unique/+entities require base strategy", func(t *testing.T) {

95 + tests := []string{

96 + "unique", // modifier alone

97 + "entities", // modifier alone

98 + "unique+entities", // modifiers without base

99 + "roots+unique", // roots is incompatible

100 + "roots+entities", // roots is incompatible

101 + "roots+pinned+unique", // roots mixed with pinned+unique

102 + }

103 +

104 + for _, input := range tests {

105 + _, err := ParseProvideStrategy(input)

106 + require.Error(t, err, "ParseProvideStrategy(%q) should fail", input)

107 + }

108 + })

109 +}

110 +

111 +func TestMustParseProvideStrategy(t *testing.T) {

112 + t.Run("valid input returns strategy", func(t *testing.T) {

113 + assert.Equal(t, ProvideStrategyAll, MustParseProvideStrategy("all"))

114 + assert.Equal(t, ProvideStrategyPinned|ProvideStrategyMFS, MustParseProvideStrategy("pinned+mfs"))

115 + })

116 +

117 + t.Run("invalid input panics", func(t *testing.T) {

118 + assert.Panics(t, func() { MustParseProvideStrategy("bogus") })

119 + assert.Panics(t, func() { MustParseProvideStrategy("all+pinned") })

120 + })

121 +}

122 +

123 +func TestValidateProvideConfig_Strategy(t *testing.T) {

124 + t.Run("valid strategies", func(t *testing.T) {

125 + for _, s := range []string{

126 + "all", "pinned", "roots", "mfs", "pinned+mfs",

127 + "pinned+unique", "pinned+entities", "pinned+mfs+entities",

128 + } {

129 + cfg := &Provide{Strategy: NewOptionalString(s)}

130 + require.NoError(t, ValidateProvideConfig(cfg), "strategy=%q", s)

131 + }

132 + })

133 +

134 + t.Run("default (nil) strategy is valid", func(t *testing.T) {

135 + cfg := &Provide{}

136 + require.NoError(t, ValidateProvideConfig(cfg))

137 + })

138 +

139 + t.Run("invalid strategy", func(t *testing.T) {

140 + cfg := &Provide{Strategy: NewOptionalString("bogus")}

141 + err := ValidateProvideConfig(cfg)

142 + require.Error(t, err)

143 + assert.Contains(t, err.Error(), "Provide.Strategy")

144 + })

145 +

146 + t.Run("all combined with others", func(t *testing.T) {

147 + cfg := &Provide{Strategy: NewOptionalString("all+pinned")}

148 + err := ValidateProvideConfig(cfg)

149 + require.Error(t, err)

150 + assert.Contains(t, err.Error(), "cannot be combined")

151 + })

152 }

153

154 func TestValidateProvideConfig_Interval(t *testing.T) {

@@ -70,6 +189,49 @@ func TestValidateProvideConfig_Interval(t *testing.T) {

189 }

190 }

191

192 +func TestValidateProvideConfig_BloomFPRate(t *testing.T) {

193 + tests := []struct {

194 + name string

195 + fpRate int64

196 + wantErr bool

197 + errMsg string

198 + }{

199 + {"valid default value", DefaultProvideBloomFPRate, false, ""},

200 + {"valid minimum (1M)", MinProvideBloomFPRate, false, ""},

201 + {"valid high (10M)", 10_000_000, false, ""},

202 + {"valid very high (100M)", 100_000_000, false, ""},

203 + {"invalid below minimum (999_999)", 999_999, true, "must be >="},

204 + {"invalid small (10_000)", 10_000, true, "must be >="},

205 + {"invalid one", 1, true, "must be >="},

206 + {"invalid zero", 0, true, "must be >="},

207 + {"invalid negative", -1, true, "must be >="},

208 + }

209 +

210 + for _, tt := range tests {

211 + t.Run(tt.name, func(t *testing.T) {

212 + cfg := &Provide{

213 + BloomFPRate: NewOptionalInteger(tt.fpRate),

214 + }

215 +

216 + err := ValidateProvideConfig(cfg)

217 +

218 + if tt.wantErr {

219 + require.Error(t, err, "expected error for fpRate=%d", tt.fpRate)

220 + if tt.errMsg != "" {

221 + assert.Contains(t, err.Error(), tt.errMsg, "error message mismatch")

222 + }

223 + } else {

224 + require.NoError(t, err, "unexpected error for fpRate=%d", tt.fpRate)

225 + }

226 + })

227 + }

228 +

229 + t.Run("default (nil) BloomFPRate is valid", func(t *testing.T) {

230 + cfg := &Provide{}

231 + require.NoError(t, ValidateProvideConfig(cfg))

232 + })

233 +}

234 +

235 func TestValidateProvideConfig_MaxWorkers(t *testing.T) {

236 tests := []struct {

237 name string

core/commands/add.go

+65 -23

@@ -19,6 +19,7 @@ import (

19 mfs "github.com/ipfs/boxo/mfs"

20 "github.com/ipfs/boxo/path"

21 "github.com/ipfs/boxo/verifcid"

22 + cid "github.com/ipfs/go-cid"

23 cmds "github.com/ipfs/go-ipfs-cmds"

24 ipld "github.com/ipfs/go-ipld-format"

25 coreiface "github.com/ipfs/kubo/core/coreiface"

@@ -68,6 +69,7 @@ const (

69 mtimeOptionName = "mtime"

70 mtimeNsecsOptionName = "mtime-nsecs"

71 fastProvideRootOptionName = "fast-provide-root"

72 + fastProvideDAGOptionName = "fast-provide-dag"

73 fastProvideWaitOptionName = "fast-provide-wait"

74 emptyDirsOptionName = "empty-dirs"

75 )

@@ -82,26 +84,41 @@ var AddCmd = &cmds.Command{

84 ShortDescription: `

85 Adds the content of <path> to IPFS. Use -r to add directories (recursively).

85 -FAST PROVIDE OPTIMIZATION:

87 +CONTENT DISCOVERABILITY:

87 -When you add content to IPFS, the sweep provider queues it for efficient

88 -DHT provides over time. While this is resource-efficient, other peers won't

89 -find your content immediately after 'ipfs add' completes.

89 +How quickly other peers can find your content depends on Provide.Strategy:

91 -To make sharing faster, 'ipfs add' does an immediate provide of the root CID

92 -to the DHT in addition to the regular queue. This complements the sweep provider:

93 -fast-provide handles the urgent case (root CIDs that users share and reference),

94 -while the sweep provider efficiently provides all blocks according to

95 -Provide.Strategy over time.

91 + Provide.Strategy=all (default):

92 + Every block is announced to the routing system as it is written to

93 + the blockstore. Content is discoverable immediately.

97 -By default, this immediate provide runs in the background without blocking

98 -the command. If you need certainty that the root CID is discoverable before

99 -the command returns (e.g., sharing a link immediately), use --fast-provide-wait

100 -to wait for the provide to complete. Use --fast-provide-root=false to skip

101 -this optimization.

95 + Selective strategies (pinned, mfs, pinned+mfs):

96 + Only the root CID is announced immediately after 'ipfs add'.

97 + Remaining blocks are announced during the next reprovide cycle

98 + (Provide.DHT.Interval, default 22h).

103 -This works best with the sweep provider and accelerated DHT client.

104 -Automatically skipped when DHT is not available.

100 +FAST PROVIDE FLAGS:

101 +

102 + --fast-provide-root (default: enabled)

103 + Announce the root CID to the routing system immediately after add,

104 + in addition to the regular provide queue. Runs in the background

105 + without blocking. Set to false to skip extra provides and minimize

106 + network overhead when importing a lot of data at once.

107 +

108 + --fast-provide-dag (default: disabled)

109 + Walk and provide the full DAG immediately after add, using the

110 + active Provide.Strategy to determine scope. Useful with selective

111 + strategies when all blocks need to be discoverable right away.

112 + No effect with Provide.Strategy=all (blockstore already provides

113 + every block on write).

114 +

115 + --fast-provide-wait (default: disabled)

116 + Block until the immediate provide completes before returning.

117 + Use when you need certainty that content is discoverable before

118 + the command returns (e.g., sharing a link immediately after adding).

119 +

120 +All fast-provide flags require an active DHT client. Skipped automatically

121 +when only HTTP delegated routing is configured.

122 `,

123 LongDescription: `

124 Adds the content of <path> to IPFS. Use -r to add directories.

@@ -265,6 +282,7 @@ https://github.com/ipfs/kubo/blob/master/docs/config.md#import

282 cmds.Int64Option(mtimeOptionName, "Custom POSIX modification time to store in created UnixFS entries (seconds before or after the Unix Epoch). WARNING: experimental, forces dag-pb for root block, disables raw-leaves"),

283 cmds.UintOption(mtimeNsecsOptionName, "Custom POSIX modification time (optional time fraction in nanoseconds)"),

284 cmds.BoolOption(fastProvideRootOptionName, "Immediately provide root CID to DHT in addition to regular queue, for faster discovery. Default: Import.FastProvideRoot"),

285 + cmds.BoolOption(fastProvideDAGOptionName, "Walk and provide the full DAG according to Provide.Strategy immediately after add. Default: Import.FastProvideDAG"),

286 cmds.BoolOption(fastProvideWaitOptionName, "Block until the immediate provide completes before returning. Default: Import.FastProvideWait"),

287 },

288 PreRun: func(req *cmds.Request, env cmds.Environment) error {

@@ -338,6 +356,7 @@ https://github.com/ipfs/kubo/blob/master/docs/config.md#import

356 mtime, _ := req.Options[mtimeOptionName].(int64)

357 mtimeNsecs, _ := req.Options[mtimeNsecsOptionName].(uint)

358 fastProvideRoot, fastProvideRootSet := req.Options[fastProvideRootOptionName].(bool)

359 + fastProvideDAG, fastProvideDAGSet := req.Options[fastProvideDAGOptionName].(bool)

360 fastProvideWait, fastProvideWaitSet := req.Options[fastProvideWaitOptionName].(bool)

361 emptyDirs, _ := req.Options[emptyDirsOptionName].(bool)

362

@@ -390,8 +409,17 @@ https://github.com/ipfs/kubo/blob/master/docs/config.md#import

409 sizeEstimationMode = cfg.Import.HAMTSizeEstimationMode()

410

411 fastProvideRoot = config.ResolveBoolFromConfig(fastProvideRoot, fastProvideRootSet, cfg.Import.FastProvideRoot, config.DefaultFastProvideRoot)

412 + fastProvideDAG = config.ResolveBoolFromConfig(fastProvideDAG, fastProvideDAGSet, cfg.Import.FastProvideDAG, config.DefaultFastProvideDAG)

413 fastProvideWait = config.ResolveBoolFromConfig(fastProvideWait, fastProvideWaitSet, cfg.Import.FastProvideWait, config.DefaultFastProvideWait)

414

415 + // --only-hash does not store data, so pinning and providing

416 + // are meaningless.

417 + if onlyHash {

418 + dopin = false

419 + fastProvideRoot = false

420 + fastProvideDAG = false

421 + }

422 +

423 // Storing optional mode or mtime (UnixFS 1.5) requires root block

424 // to always be 'dag-pb' and not 'raw'. Below adjusts raw-leaves setting, if possible.

425 if preserveMode || preserveMtime || mode != 0 || mtime != 0 {

@@ -642,20 +670,34 @@ https://github.com/ipfs/kubo/blob/master/docs/config.md#import

670 return fmt.Errorf("expected a file argument")

671 }

672

645 - // Apply fast-provide-root if the flag is enabled

646 - if fastProvideRoot && (lastRootCid != path.ImmutablePath{}) {

673 + hasRoot := lastRootCid != path.ImmutablePath{}

674 +

675 + if fastProvideDAG && hasRoot {

676 + // DAG walk includes the root CID (DFS pre-order emits it

677 + // first), so a separate root provide is not needed.

678 + cmdenv.ExecuteFastProvideDAG(

679 + req.Context,

680 + ipfsNode.Context(),

681 + []cid.Cid{lastRootCid.RootCid()},

682 + ipfsNode.ProvidingStrategy,

683 + ipfsNode.Blockstore,

684 + ipfsNode.Provider,

685 + fastProvideWait,

686 + uint(cfg.Provide.BloomFPRate.WithDefault(config.DefaultProvideBloomFPRate)),

687 + 0, // block count unknown here; bloom chain auto-grows

688 + )

689 + } else if fastProvideRoot && hasRoot {

690 cfg, err := ipfsNode.Repo.Config()

691 if err != nil {

692 return err

693 }

651 - if err := cmdenv.ExecuteFastProvide(req.Context, ipfsNode, cfg, lastRootCid.RootCid(), fastProvideWait, dopin, dopin, toFilesSet); err != nil {

694 + if err := cmdenv.ExecuteFastProvideRoot(req.Context, ipfsNode, cfg, lastRootCid.RootCid(), fastProvideWait, dopin, dopin, toFilesSet); err != nil {

695 return err

696 }

654 - } else if !fastProvideRoot {

697 + } else if !fastProvideRoot && !fastProvideDAG {

698 + log.Debugw("fast-provide-root: skipped", "reason", "disabled by flag or config")

699 if fastProvideWait {

656 - log.Debugw("fast-provide-root: skipped", "reason", "disabled by flag or config", "wait-flag-ignored", true)

657 - } else {

658 - log.Debugw("fast-provide-root: skipped", "reason", "disabled by flag or config")

700 + log.Debugw("fast-provide-root: wait-flag-ignored")

701 }

702 }

703

core/commands/cmdenv/env.go

+116 -8

@@ -6,16 +6,18 @@ import (

6 "strconv"

7 "strings"

9 + "github.com/ipfs/boxo/blockstore"

10 + "github.com/ipfs/boxo/dag/walker"

11 "github.com/ipfs/go-cid"

12 cmds "github.com/ipfs/go-ipfs-cmds"

13 logging "github.com/ipfs/go-log/v2"

12 - routing "github.com/libp2p/go-libp2p/core/routing"

13 -

14 "github.com/ipfs/kubo/commands"

15 "github.com/ipfs/kubo/config"

16 "github.com/ipfs/kubo/core"

17 coreiface "github.com/ipfs/kubo/core/coreiface"

18 options "github.com/ipfs/kubo/core/coreiface/options"

19 + "github.com/ipfs/kubo/core/node"

20 + routing "github.com/libp2p/go-libp2p/core/routing"

21 )

23 var log = logging.Logger("core/commands/cmdenv")

@@ -107,7 +109,7 @@ func provideCIDSync(ctx context.Context, router routing.Routing, c cid.Cid) erro

109 return router.Provide(ctx, c, true)

110 }

111

110 -// ExecuteFastProvide immediately provides a root CID to the DHT, bypassing the regular

112 +// ExecuteFastProvideRoot immediately provides a root CID to the DHT, bypassing the regular

113 // provide queue for faster content discovery. This function is reusable across commands

114 // that add or import content, such as ipfs add and ipfs dag import.

115 //

@@ -129,7 +131,7 @@ func provideCIDSync(ctx context.Context, router routing.Routing, c cid.Cid) erro

131 // The function handles all precondition checks (Provide.Enabled, DHT availability,

132 // strategy matching) and logs appropriately. In async mode, it launches a goroutine

133 // with a detached context and timeout.

132 -func ExecuteFastProvide(

134 +func ExecuteFastProvideRoot(

135 ctx context.Context,

136 ipfsNode *core.IpfsNode,

137 cfg *config.Config,

@@ -156,7 +158,7 @@ func ExecuteFastProvide(

158

159 // Check if strategy allows providing this content

160 strategyStr := cfg.Provide.Strategy.WithDefault(config.DefaultProvideStrategy)

159 - strategy := config.ParseProvideStrategy(strategyStr)

161 + strategy := config.MustParseProvideStrategy(strategyStr)

162 shouldProvide := config.ShouldProvideForStrategy(strategy, isPinned, isPinnedRoot, isMFS)

163

164 if !shouldProvide {

@@ -176,11 +178,14 @@ func ExecuteFastProvide(

178 return nil

179 }

180

179 - // Asynchronous mode (default): fire-and-forget, don't block, always return nil

181 + // Asynchronous mode (default): fire-and-forget, don't block, always return nil.

182 + // Parent off the node's lifetime context (not context.Background) so the

183 + // goroutine cancels on daemon shutdown instead of potentially outliving

184 + // the node and touching a closed DHT client. The timeout still bounds

185 + // stuck DHT operations.

186 log.Debugw("fast-provide-root: providing asynchronously", "cid", rootCid)

187 go func() {

182 - // Use detached context with timeout to prevent hanging on network issues

183 - ctx, cancel := context.WithTimeout(context.Background(), config.DefaultFastProvideTimeout)

188 + ctx, cancel := context.WithTimeout(ipfsNode.Context(), config.DefaultFastProvideTimeout)

189 defer cancel()

190 if err := provideCIDSync(ctx, ipfsNode.DHTClient, rootCid); err != nil {

191 log.Warnw("fast-provide-root: async provide failed", "cid", rootCid, "error", err)

@@ -190,3 +195,106 @@ func ExecuteFastProvide(

195 }()

196 return nil

197 }

198 +

199 +// ExecuteFastProvideDAG walks the DAGs rooted at roots and provides

200 +// CIDs according to the active Provide.Strategy. A single bloom

201 +// tracker is shared across all roots so shared sub-DAGs are

202 +// deduplicated. Uses an unbuffered channel for backpressure.

203 +//

204 +// Context handling:

205 +// - wait=true: the walk runs inline under cmdCtx (the request

206 +// context), so a user Ctrl+C on the command cancels the walk.

207 +// - wait=false: the walk runs in a background goroutine under

208 +// nodeCtx (the IpfsNode lifetime context). This lets the walk

209 +// survive the command handler returning (go-ipfs-cmds cancels

210 +// req.Context on handler exit) while still being cancelled on

211 +// daemon shutdown, so the goroutine does not outlive the node

212 +// and keep the blockstore/provider pinned open.

213 +//

214 +// fpRate is the bloom filter target false-positive rate (1/N), normally

215 +// resolved from cfg.Provide.BloomFPRate by the caller.

216 +// blockCount sizes the bloom filter (pass 0 if unknown).

217 +func ExecuteFastProvideDAG(

218 + cmdCtx context.Context,

219 + nodeCtx context.Context,

220 + roots []cid.Cid,

221 + strategy config.ProvideStrategy,

222 + bs blockstore.Blockstore,

223 + prov node.DHTProvider,

224 + wait bool,

225 + fpRate uint,

226 + blockCount uint,

227 +) {

228 + if len(roots) == 0 {

229 + return

230 + }

231 + if (strategy&config.ProvideStrategyPinned) == 0 &&

232 + (strategy&config.ProvideStrategyMFS) == 0 {

233 + return

234 + }

235 +

236 + do := func(ctx context.Context) {

237 + expectedItems := max(uint(walker.DefaultBloomInitialCapacity), blockCount)

238 + tracker, err := walker.NewBloomTracker(expectedItems, fpRate)

239 + if err != nil {

240 + log.Errorf("fast-provide-dag: bloom tracker: %s", err)

241 + return

242 + }

243 +

244 + ch := make(chan cid.Cid) // unbuffered for backpressure

245 + done := make(chan struct{})

246 + go func() {

247 + defer close(done)

248 + for c := range ch {

249 + if err := prov.StartProviding(false, c.Hash()); err != nil {

250 + log.Errorf("fast-provide-dag: %s: %s", c, err)

251 + }

252 + }

253 + }()

254 +

255 + emit := func(c cid.Cid) bool {

256 + select {

257 + case ch <- c:

258 + return true

259 + case <-ctx.Done():

260 + return false

261 + }

262 + }

263 +

264 + opts := []walker.Option{walker.WithVisitedTracker(tracker)}

265 + useEntities := strategy&config.ProvideStrategyEntities != 0

266 +

267 + if useEntities {

268 + fetch := walker.NodeFetcherFromBlockstore(bs)

269 + for _, root := range roots {

270 + if ctx.Err() != nil {

271 + break

272 + }

273 + _ = walker.WalkEntityRoots(ctx, root, fetch, emit, opts...)

274 + }

275 + } else {

276 + fetch := walker.LinksFetcherFromBlockstore(bs)

277 + for _, root := range roots {

278 + if ctx.Err() != nil {

279 + break

280 + }

281 + _ = walker.WalkDAG(ctx, root, fetch, emit, opts...)

282 + }

283 + }

284 +

285 + close(ch)

286 + <-done

287 + log.Infow("fast-provide-dag: finished",

288 + "providedCIDs", tracker.Count(),

289 + "skippedBranches", tracker.Deduplicated())

290 + }

291 +

292 + if wait {

293 + do(cmdCtx)

294 + } else {

295 + // Use the node's lifetime context so the walk survives

296 + // the command handler returning (which cancels req.Context)

297 + // but still cancels on daemon shutdown.

298 + go do(nodeCtx)

299 + }

300 +}

core/commands/dag/dag.go

@@ -22,6 +22,7 @@ const (

22 silentOptionName = "silent"

23 statsOptionName = "stats"

24 fastProvideRootOptionName = "fast-provide-root"

25 + fastProvideDAGOptionName = "fast-provide-dag"

26 fastProvideWaitOptionName = "fast-provide-wait"

27 )

@@ -216,6 +217,7 @@ Specification of CAR formats: https://ipld.io/specs/transport/car/

217 cmds.BoolOption(silentOptionName, "No output."),

218 cmds.BoolOption(statsOptionName, "Output stats."),

219 cmds.BoolOption(fastProvideRootOptionName, "Immediately provide root CIDs to DHT in addition to regular queue, for faster discovery. Default: Import.FastProvideRoot"),

220 + cmds.BoolOption(fastProvideDAGOptionName, "Walk and provide the full DAG according to Provide.Strategy after import. Default: Import.FastProvideDAG"),

221 cmds.BoolOption(fastProvideWaitOptionName, "Block until the immediate provide completes before returning. Default: Import.FastProvideWait"),

222 cmdutils.AllowBigBlockOption,

223 },

core/commands/dag/import.go

+24 -8

@@ -51,9 +51,11 @@ func dagImport(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment

51 doPinRoots, _ := req.Options[pinRootsOptionName].(bool)

53 fastProvideRoot, fastProvideRootSet := req.Options[fastProvideRootOptionName].(bool)

54 + fastProvideDAG, fastProvideDAGSet := req.Options[fastProvideDAGOptionName].(bool)

55 fastProvideWait, fastProvideWaitSet := req.Options[fastProvideWaitOptionName].(bool)

57 fastProvideRoot = config.ResolveBoolFromConfig(fastProvideRoot, fastProvideRootSet, cfg.Import.FastProvideRoot, config.DefaultFastProvideRoot)

58 + fastProvideDAG = config.ResolveBoolFromConfig(fastProvideDAG, fastProvideDAGSet, cfg.Import.FastProvideDAG, config.DefaultFastProvideDAG)

59 fastProvideWait = config.ResolveBoolFromConfig(fastProvideWait, fastProvideWaitSet, cfg.Import.FastProvideWait, config.DefaultFastProvideWait)

61 // grab a pinlock ( which doubles as a GC lock ) so that regardless of the

@@ -210,20 +212,34 @@ func dagImport(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment

212 }

213 }

214

213 - // Fast-provide roots for faster discovery

214 - if fastProvideRoot {

215 + // Provide imported content for faster discovery.

216 + // DAG walk supersedes root-only (root is included in the walk).

217 + if fastProvideDAG {

218 + var rootCIDs []cid.Cid

219 + _ = roots.ForEach(func(c cid.Cid) error {

220 + rootCIDs = append(rootCIDs, c)

221 + return nil

222 + })

223 + cmdenv.ExecuteFastProvideDAG(

224 + req.Context,

225 + node.Context(),

226 + rootCIDs,

227 + node.ProvidingStrategy,

228 + node.Blockstore,

229 + node.Provider,

230 + fastProvideWait,

231 + uint(cfg.Provide.BloomFPRate.WithDefault(config.DefaultProvideBloomFPRate)),

232 + 0, // block count unknown; bloom chain auto-grows

233 + )

234 + } else if fastProvideRoot {

235 err = roots.ForEach(func(c cid.Cid) error {

216 - return cmdenv.ExecuteFastProvide(req.Context, node, cfg, c, fastProvideWait, doPinRoots, doPinRoots, false)

236 + return cmdenv.ExecuteFastProvideRoot(req.Context, node, cfg, c, fastProvideWait, doPinRoots, doPinRoots, false)

237 })

238 if err != nil {

239 return err

240 }

241 } else {

222 - if fastProvideWait {

223 - log.Debugw("fast-provide-root: skipped", "reason", "disabled by flag or config", "wait-flag-ignored", true)

224 - } else {

225 - log.Debugw("fast-provide-root: skipped", "reason", "disabled by flag or config")

226 - }

242 + log.Debugw("fast-provide-root: skipped", "reason", "disabled by flag or config")

243 }

244

245 return nil

core/commands/pin/pin.go

+91 -2

@@ -20,6 +20,7 @@ import (

20 coreiface "github.com/ipfs/kubo/core/coreiface"

21 options "github.com/ipfs/kubo/core/coreiface/options"

23 + config "github.com/ipfs/kubo/config"

24 core "github.com/ipfs/kubo/core"

25 cmdenv "github.com/ipfs/kubo/core/commands/cmdenv"

26 "github.com/ipfs/kubo/core/commands/cmdutils"

@@ -52,8 +53,11 @@ type AddPinOutput struct {

53 }

55 const (

55 - pinRecursiveOptionName = "recursive"

56 - pinProgressOptionName = "progress"

56 + pinRecursiveOptionName = "recursive"

57 + pinProgressOptionName = "progress"

58 + fastProvideRootOptionName = "fast-provide-root"

59 + fastProvideDAGOptionName = "fast-provide-dag"

60 + fastProvideWaitOptionName = "fast-provide-wait"

61 )

63 var addPinCmd = &cmds.Command{

@@ -89,6 +93,9 @@ It may take some time. Pass '--progress' to track the progress.

93 cmds.BoolOption(pinRecursiveOptionName, "r", "Recursively pin the object linked to by the specified object(s).").WithDefault(true),

94 cmds.StringOption(pinNameOptionName, "n", "An optional name for created pin(s)."),

95 cmds.BoolOption(pinProgressOptionName, "Show progress"),

96 + cmds.BoolOption(fastProvideRootOptionName, "Immediately provide root CID to DHT after pinning. Default: Import.FastProvideRoot"),

97 + cmds.BoolOption(fastProvideDAGOptionName, "Walk and provide the full DAG according to Provide.Strategy after pinning. Default: Import.FastProvideDAG"),

98 + cmds.BoolOption(fastProvideWaitOptionName, "Block until the immediate provide completes. Default: Import.FastProvideWait"),

99 },

100 Type: AddPinOutput{},

101 Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) error {

@@ -116,12 +123,15 @@ It may take some time. Pass '--progress' to track the progress.

123 return err

124 }

125

126 + nd, fpRoot, fpDAG, fpWait := resolveFastProvideFlags(req, env)

127 +

128 if !showProgress {

129 added, err := pinAddMany(req.Context, api, enc, req.Arguments, recursive, name)

130 if err != nil {

131 return err

132 }

133

134 + fastProvideAfterPin(req, nd, fpRoot, fpDAG, fpWait, added)

135 return cmds.EmitOnce(res, &AddPinOutput{Pins: added})

136 }

137

@@ -149,6 +159,8 @@ It may take some time. Pass '--progress' to track the progress.

159 return val.err

160 }

161

162 + fastProvideAfterPin(req, nd, fpRoot, fpDAG, fpWait, val.pins)

163 +

164 if ps := v.ProgressStat(); ps.Nodes != 0 {

165 if err := res.Emit(&AddPinOutput{Progress: ps.Nodes, Bytes: ps.Bytes}); err != nil {

166 return err

@@ -234,6 +246,77 @@ func pinAddMany(ctx context.Context, api coreiface.CoreAPI, enc cidenc.Encoder,

246 return added, nil

247 }

248

249 +// resolveFastProvideFlags resolves --fast-provide-root, --fast-provide-dag,

250 +// and --fast-provide-wait from CLI flags, falling back to config defaults.

251 +// Returns the node for use by fastProvideAfterPin.

252 +func resolveFastProvideFlags(req *cmds.Request, env cmds.Environment) (nd *core.IpfsNode, root, dag, wait bool) {

253 + nd, err := cmdenv.GetNode(env)

254 + if err != nil {

255 + return nil, config.DefaultFastProvideRoot, config.DefaultFastProvideDAG, config.DefaultFastProvideWait

256 + }

257 + cfg, err := nd.Repo.Config()

258 + if err != nil {

259 + return nd, config.DefaultFastProvideRoot, config.DefaultFastProvideDAG, config.DefaultFastProvideWait

260 + }

261 + fpRoot, fpRootSet := req.Options[fastProvideRootOptionName].(bool)

262 + fpDAG, fpDAGSet := req.Options[fastProvideDAGOptionName].(bool)

263 + fpWait, fpWaitSet := req.Options[fastProvideWaitOptionName].(bool)

264 + root = config.ResolveBoolFromConfig(fpRoot, fpRootSet, cfg.Import.FastProvideRoot, config.DefaultFastProvideRoot)

265 + dag = config.ResolveBoolFromConfig(fpDAG, fpDAGSet, cfg.Import.FastProvideDAG, config.DefaultFastProvideDAG)

266 + wait = config.ResolveBoolFromConfig(fpWait, fpWaitSet, cfg.Import.FastProvideWait, config.DefaultFastProvideWait)

267 + return nd, root, dag, wait

268 +}

269 +

270 +// fastProvideAfterPin handles both root and DAG providing after a

271 +// successful pin operation. Best-effort: errors are logged but do not

272 +// fail the pin command.

273 +func fastProvideAfterPin(req *cmds.Request, nd *core.IpfsNode, fpRoot, fpDAG, fpWait bool, encodedCIDs []string) {

274 + if !fpRoot && !fpDAG {

275 + return

276 + }

277 + cfg, err := nd.Repo.Config()

278 + if err != nil {

279 + return

280 + }

281 + var cidList []cid.Cid

282 + for _, s := range encodedCIDs {

283 + c, err := cid.Decode(s)

284 + if err != nil {

285 + continue

286 + }

287 + cidList = append(cidList, c)

288 + }

289 +

290 + if fpDAG {

291 + // DAG walk includes the root CID (DFS pre-order emits it

292 + // first), so a separate root provide is not needed.

293 + // Single call with all roots shares one bloom tracker.

294 + cmdenv.ExecuteFastProvideDAG(

295 + req.Context,

296 + nd.Context(),

297 + cidList,

298 + nd.ProvidingStrategy,

299 + nd.Blockstore,

300 + nd.Provider,

301 + fpWait,

302 + uint(cfg.Provide.BloomFPRate.WithDefault(config.DefaultProvideBloomFPRate)),

303 + 0, // block count unknown; bloom chain auto-grows

304 + )

305 + } else if fpRoot {

306 + for _, c := range cidList {

307 + if err := cmdenv.ExecuteFastProvideRoot(

308 + req.Context, nd, cfg, c,

309 + fpWait,

310 + true, // isPinned

311 + true, // isPinnedRoot

312 + false, // isMFS

313 + ); err != nil {

314 + log.Errorf("fast provide root after pin: %s", err)

315 + }

316 + }

317 + }

318 +}

319 +

320 var rmPinCmd = &cmds.Command{

321 Helptext: cmds.HelpText{

322 Tagline: "Remove object from pin-list.",

@@ -622,6 +705,9 @@ pin.

705 },

706 Options: []cmds.Option{

707 cmds.BoolOption(pinUnpinOptionName, "Remove the old pin.").WithDefault(true),

708 + cmds.BoolOption(fastProvideRootOptionName, "Immediately provide new root CID to DHT after update. Default: Import.FastProvideRoot"),

709 + cmds.BoolOption(fastProvideDAGOptionName, "Walk and provide the full DAG according to Provide.Strategy after update. Default: Import.FastProvideDAG"),

710 + cmds.BoolOption(fastProvideWaitOptionName, "Block until the immediate provide completes. Default: Import.FastProvideWait"),

711 },

712 Type: PinOutput{},

713 Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) error {

@@ -662,6 +748,9 @@ pin.

748 return err

749 }

750

751 + nd, fpRoot, fpDAG, fpWait := resolveFastProvideFlags(req, env)

752 + fastProvideAfterPin(req, nd, fpRoot, fpDAG, fpWait, []string{enc.Encode(to.RootCid())})

753 +

754 return cmds.EmitOnce(res, &PinOutput{Pins: []string{enc.Encode(from.RootCid()), enc.Encode(to.RootCid())}})

755 },

756 Encoders: cmds.EncoderMap{

core/commands/routing.go

+10 -4

@@ -269,11 +269,16 @@ var provideRefRoutingCmd = &cmds.Command{

269 }

270

271 var reprovideRoutingCmd = &cmds.Command{

272 - Status: cmds.Experimental,

272 + Status: cmds.Deprecated,

273 Helptext: cmds.HelpText{

274 - Tagline: "Trigger reprovider.",

274 + Tagline: "Trigger reprovider (legacy provider only).",

275 ShortDescription: `

276 Trigger reprovider to announce our data to network.

277 +

278 +Only available with the legacy provider (Provide.DHT.SweepEnabled=false).

279 +Returns an error when Provide.DHT.SweepEnabled=true (the default).

280 +The sweep provider reprovides automatically on schedule.

281 +Use 'ipfs provide stat -a' to monitor reprovide progress.

282 `,

283 },

284 Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) error {

@@ -286,7 +291,6 @@ Trigger reprovider to announce our data to network.

291 return ErrNotOnline

292 }

293

289 - // respect global config

294 cfg, err := nd.Repo.Config()

295 if err != nil {

296 return err

@@ -299,7 +303,9 @@ Trigger reprovider to announce our data to network.

303 }

304 provideSys, ok := nd.Provider.(provider.Reprovider)

305 if !ok {

302 - return errors.New("manual reprovide only available with legacy provider (Provide.DHT.SweepEnabled=false)")

306 + err := errors.New("invalid configuration: manual reprovide not available with sweep provider (Provide.DHT.SweepEnabled=true), use 'ipfs provide stat -a' to monitor automatic reprovide progress")

307 + log.Error(err)

308 + return err

309 }

310

311 err = provideSys.Reprovide(req.Context)

core/coreapi/coreapi.go

+2 -4

@@ -70,8 +70,7 @@ type CoreAPI struct {

70 ipldPathResolver pathresolver.Resolver

71 unixFSPathResolver pathresolver.Resolver

73 - provider node.DHTProvider

74 - providingStrategy config.ProvideStrategy

73 + provider node.DHTProvider

75 pubSub *pubsub.PubSub

@@ -186,8 +185,7 @@ func (api *CoreAPI) WithOptions(opts ...options.ApiOption) (coreiface.CoreAPI, e

185 ipldPathResolver: n.IPLDPathResolver,

186 unixFSPathResolver: n.UnixFSPathResolver,

187

189 - provider: n.Provider,

190 - providingStrategy: n.ProvidingStrategy,

188 + provider: n.Provider,

189

190 pubSub: n.PubSub,

191

core/coreapi/unixfs.go

+17 -54

@@ -16,7 +16,6 @@ import (

16 uio "github.com/ipfs/boxo/ipld/unixfs/io"

17 "github.com/ipfs/boxo/mfs"

18 "github.com/ipfs/boxo/path"

19 - "github.com/ipfs/boxo/provider"

19 cid "github.com/ipfs/go-cid"

20 cidutil "github.com/ipfs/go-cidutil"

21 ds "github.com/ipfs/go-datastore"

@@ -28,7 +27,6 @@ import (

27 options "github.com/ipfs/kubo/core/coreiface/options"

28 "github.com/ipfs/kubo/core/coreunix"

29 "github.com/ipfs/kubo/tracing"

31 - mh "github.com/multiformats/go-multihash"

30 "go.opentelemetry.io/otel/attribute"

31 "go.opentelemetry.io/otel/trace"

32 )

@@ -110,19 +108,21 @@ func (api *UnixfsAPI) Add(ctx context.Context, files files.Node, opts ...options

108

109 var dserv ipld.DAGService = merkledag.NewDAGService(bserv)

110

113 - // wrap the DAGService in a providingDAG service which provides every block written.

114 - // note about strategies:

115 - // - "all" gets handled directly at the blockstore so no need to provide

116 - // - "roots" gets handled in the pinner

117 - // - "mfs" gets handled in mfs

118 - // We need to provide the "pinned" cases only. Added blocks are not

119 - // going to be provided by the blockstore (wrong strategy for that),

120 - // nor by the pinner (the pinner doesn't traverse the pinned DAG itself, it only

121 - // handles roots). This wrapping ensures all blocks of pinned content get provided.

122 - if settings.Pin && !settings.OnlyHash &&

123 - (api.providingStrategy&config.ProvideStrategyPinned) != 0 {

124 - dserv = &providingDagService{dserv, api.provider}

125 - }

111 + // Per-block providing for new content is handled outside the add

112 + // pipeline:

113 + //

114 + // - Provide.Strategy=all: every block is provided at the

115 + // blockstore level via the blockstore.Provider hook

116 + // (see core/node/storage.go).

117 + // - Selective strategies (pinned, mfs, +unique, +entities) with

118 + // --fast-provide-dag: ExecuteFastProvideDAG walks the DAG once

119 + // after add completes, applying the active strategy and bloom

120 + // dedup. Wiring lives in core/commands/add.go.

121 + // - --fast-provide-root only (default): the root CID is announced

122 + // immediately via ExecuteFastProvideRoot in the command handler.

123 + //

124 + // The coreapi layer therefore does not wrap the DAGService with

125 + // any providing logic.

126

127 // add a sync call to the DagService

128 // this ensures that data written to the DagService is persisted to the underlying datastore

@@ -147,9 +147,8 @@ func (api *UnixfsAPI) Add(ctx context.Context, files files.Node, opts ...options

147 }

148

149 // Note: the dag service gets wrapped multiple times:

150 - // 1. providingDagService (if pinned strategy) - provides blocks as they're added

151 - // 2. syncDagService - ensures data persistence

152 - // 3. batchingDagService (in coreunix.Adder) - batches operations for efficiency

150 + // 1. syncDagService - ensures data persistence

151 + // 2. batchingDagService (in coreunix.Adder) - batches operations for efficiency

152

153 fileAdder, err := coreunix.NewAdder(ctx, pinning, addblockstore, syncDserv)

154 if err != nil {

@@ -393,39 +392,3 @@ type syncDagService struct {

392 func (s *syncDagService) Sync() error {

393 return s.syncFn()

394 }

396 -

397 -type providingDagService struct {

398 - ipld.DAGService

399 - provider.MultihashProvider

400 -}

401 -

402 -func (pds *providingDagService) Add(ctx context.Context, n ipld.Node) error {

403 - if err := pds.DAGService.Add(ctx, n); err != nil {

404 - return err

405 - }

406 - // Provider errors are logged but not propagated.

407 - // We don't want DAG operations to fail due to providing issues.

408 - // The user's data is still stored successfully even if the

409 - // announcement to the routing system fails temporarily.

410 - if err := pds.StartProviding(false, n.Cid().Hash()); err != nil {

411 - log.Errorf("failed to provide new block: %s", err)

412 - }

413 - return nil

414 -}

415 -

416 -func (pds *providingDagService) AddMany(ctx context.Context, nds []ipld.Node) error {

417 - if err := pds.DAGService.AddMany(ctx, nds); err != nil {

418 - return err

419 - }

420 - keys := make([]mh.Multihash, len(nds))

421 - for i, n := range nds {

422 - keys[i] = n.Cid().Hash()

423 - }

424 - // Same error handling philosophy as Add(): log but don't fail.

425 - if err := pds.StartProviding(false, keys...); err != nil {

426 - log.Errorf("failed to provide new blocks: %s", err)

427 - }

428 - return nil

429 -}

430 -

431 -var _ ipld.DAGService = (*providingDagService)(nil)

core/node/core.go

+2 -5

@@ -52,10 +52,7 @@ func BlockService(cfg *config.Config) func(lc fx.Lifecycle, bs blockstore.Blocks

53 // Pinning creates new pinner which tells GC which blocks should be kept

54 func Pinning(strategy string) func(bstore blockstore.Blockstore, ds format.DAGService, repo repo.Repo, prov DHTProvider) (pin.Pinner, error) {

55 - // Parse strategy at function creation time (not inside the returned function)

56 - // This happens before the provider is created, which is why we pass the strategy

57 - // string and parse it here, rather than using fx-provided ProvidingStrategy.

58 - strategyFlag := config.ParseProvideStrategy(strategy)

55 + strategyFlag := config.MustParseProvideStrategy(strategy)

57 return func(bstore blockstore.Blockstore,

58 ds format.DAGService,

@@ -238,7 +235,7 @@ func Files(strategy string) func(mctx helpers.MetricsCtx, lc fx.Lifecycle, repo

235 // strategy - it ensures all MFS content gets announced as it's added or

236 // modified. For non-mfs strategies, we set provider to nil to avoid

237 // unnecessary providing.

241 - strategyFlag := config.ParseProvideStrategy(strategy)

238 + strategyFlag := config.MustParseProvideStrategy(strategy)

239 if strategyFlag&config.ProvideStrategyMFS == 0 {

240 prov = nil

241 }

core/node/libp2p/routingopt.go

+15 -1

@@ -260,16 +260,30 @@ func constructDHTRouting(mode dht.ModeOpt) RoutingOption {

260 wanOptions := []dht.Option{

261 dht.BootstrapPeers(args.BootstrapPeers...),

262 }

263 + // In stub mode, allow loopback peers in the WAN routing

264 + // table so Provide/PutValue work with ephemeral test peers.

265 + if os.Getenv("TEST_DHT_STUB") != "" {

266 + wanOptions = append(wanOptions,

267 + dht.AddressFilter(nil),

268 + dht.QueryFilter(func(_ any, _ peer.AddrInfo) bool { return true }),

269 + dht.RoutingTableFilter(func(_ any, _ peer.ID) bool { return true }),

270 + dht.RoutingTablePeerDiversityFilter(nil),

271 + )

272 + }

273 lanOptions := []dht.Option{}

274 if args.LoopbackAddressesOnLanDHT {

275 lanOptions = append(lanOptions, dht.AddressFilter(nil))

276 }

267 - return dual.New(

277 + d, err := dual.New(

278 args.Ctx, args.Host,

279 dual.DHTOption(dhtOpts...),

280 dual.WanDHTOption(wanOptions...),

281 dual.LanDHTOption(lanOptions...),

282 )

283 + if err != nil {

284 + return nil, err

285 + }

286 + return d, nil

287 }

288 }

289

core/node/provider.go

+196 -13

@@ -2,6 +2,7 @@ package node

3 import (

4 "context"

5 + "encoding/binary"

6 "errors"

7 "fmt"

8 "os"

@@ -9,6 +10,7 @@ import (

10 "time"

12 "github.com/ipfs/boxo/blockstore"

13 + "github.com/ipfs/boxo/dag/walker"

14 "github.com/ipfs/boxo/fetcher"

15 "github.com/ipfs/boxo/mfs"

16 pin "github.com/ipfs/boxo/pinning/pinner"

@@ -54,6 +56,11 @@ const (

57 // KeystoreDatastorePath is the base directory for the provider keystore datastores.

58 KeystoreDatastorePath = "provider-keystore"

59 +

60 + // reprovideLastUniqueCountKey stores the unique CID count from

61 + // the last +unique reprovide cycle, used to size the next cycle's

62 + // bloom filter.

63 + reprovideLastUniqueCountKey = "/reprovideLastUniqueCount"

64 )

66 var (

@@ -1094,13 +1101,14 @@ func OnlineProviders(provide bool, cfg *config.Config) fx.Option {

1101

1102 providerStrategy := cfg.Provide.Strategy.WithDefault(config.DefaultProvideStrategy)

1103

1097 - strategyFlag := config.ParseProvideStrategy(providerStrategy)

1098 - if strategyFlag == 0 {

1099 - return fx.Error(fmt.Errorf("provider: unknown strategy %q", providerStrategy))

1104 + if _, err := config.ParseProvideStrategy(providerStrategy); err != nil {

1105 + return fx.Error(fmt.Errorf("provider: %w", err))

1106 }

1107

1108 + bloomFPRate := uint(cfg.Provide.BloomFPRate.WithDefault(config.DefaultProvideBloomFPRate))

1109 +

1110 opts := []fx.Option{

1103 - fx.Provide(setReproviderKeyProvider(providerStrategy)),

1111 + fx.Provide(setReproviderKeyProvider(providerStrategy, bloomFPRate)),

1112 }

1113

1114 sweepEnabled := cfg.Provide.DHT.SweepEnabled.WithDefault(config.DefaultProvideDHTSweepEnabled)

@@ -1162,13 +1170,188 @@ type provStrategyOut struct {

1170 ProvidingKeyChanFunc provider.KeyChanFunc

1171 }

1172

1173 +// readLastUniqueCount reads the persisted unique CID count from the

1174 +// previous +unique reprovide cycle. Returns 0 if not found or corrupt.

1175 +func readLastUniqueCount(ds datastore.Datastore) uint64 {

1176 + val, err := ds.Get(context.Background(), datastore.NewKey(reprovideLastUniqueCountKey))

1177 + if err != nil {

1178 + return 0

1179 + }

1180 + if len(val) != 8 {

1181 + return 0

1182 + }

1183 + return binary.BigEndian.Uint64(val)

1184 +}

1185 +

1186 +// persistUniqueCount stores the unique CID count for the next cycle.

1187 +func persistUniqueCount(ds datastore.Datastore, count uint64) {

1188 + buf := make([]byte, 8)

1189 + binary.BigEndian.PutUint64(buf, count)

1190 + if err := ds.Put(context.Background(), datastore.NewKey(reprovideLastUniqueCountKey), buf); err != nil {

1191 + logger.Errorf("failed to persist unique count: %s", err)

1192 + }

1193 +}

1194 +

1195 +// walkFunc abstracts a DAG walk (WalkDAG or WalkEntityRoots) so the

1196 +// MFS provider can be parameterized without duplicating the

1197 +// flush+walk+channel boilerplate.

1198 +type walkFunc func(ctx context.Context, root cid.Cid, emit func(cid.Cid) bool, opts ...walker.Option) error

1199 +

1200 +// uniqueMFSProvider is the +unique counterpart of mfsProvider. It

1201 +// flushes the MFS root, then walks the MFS DAG with a shared

1202 +// VisitedTracker and a locality check (blockstore.Has) so only

1203 +// locally-present blocks are emitted.

1204 +func uniqueMFSProvider(mfsRoot *mfs.Root, bs blockstore.Blockstore, tracker walker.VisitedTracker) provider.KeyChanFunc {

1205 + walk := func(ctx context.Context, root cid.Cid, emit func(cid.Cid) bool, opts ...walker.Option) error {

1206 + return walker.WalkDAG(ctx, root, walker.LinksFetcherFromBlockstore(bs), emit, opts...)

1207 + }

1208 + return mfsWalkProvider(mfsRoot, bs, tracker, walk)

1209 +}

1210 +

1211 +// mfsEntityRootsProvider is the +entities counterpart. It walks with

1212 +// WalkEntityRoots, emitting only entity roots and skipping file chunks.

1213 +func mfsEntityRootsProvider(mfsRoot *mfs.Root, bs blockstore.Blockstore, tracker walker.VisitedTracker) provider.KeyChanFunc {

1214 + walk := func(ctx context.Context, root cid.Cid, emit func(cid.Cid) bool, opts ...walker.Option) error {

1215 + return walker.WalkEntityRoots(ctx, root, walker.NodeFetcherFromBlockstore(bs), emit, opts...)

1216 + }

1217 + return mfsWalkProvider(mfsRoot, bs, tracker, walk)

1218 +}

1219 +

1220 +// mfsWalkProvider builds a KeyChanFunc that flushes MFS, then walks

1221 +// with the given walkFunc using a shared tracker and locality check.

1222 +func mfsWalkProvider(mfsRoot *mfs.Root, bs blockstore.Blockstore, tracker walker.VisitedTracker, walk walkFunc) provider.KeyChanFunc {

1223 + return func(ctx context.Context) (<-chan cid.Cid, error) {

1224 + if err := mfsRoot.FlushMemFree(ctx); err != nil {

1225 + return nil, fmt.Errorf("provider: error flushing MFS: %w", err)

1226 + }

1227 + rootNode, err := mfsRoot.GetDirectory().GetNode()

1228 + if err != nil {

1229 + return nil, fmt.Errorf("provider: error loading MFS root: %w", err)

1230 + }

1231 +

1232 + ch := make(chan cid.Cid)

1233 + go func() {

1234 + defer close(ch)

1235 + locality := func(ctx context.Context, c cid.Cid) (bool, error) {

1236 + return bs.Has(ctx, c)

1237 + }

1238 + _ = walk(ctx, rootNode.Cid(), func(c cid.Cid) bool {

1239 + select {

1240 + case ch <- c:

1241 + return true

1242 + case <-ctx.Done():

1243 + return false

1244 + }

1245 + }, walker.WithVisitedTracker(tracker), walker.WithLocality(locality))

1246 + }()

1247 + return ch, nil

1248 + }

1249 +}

1250 +

1251 // createKeyProvider creates the appropriate KeyChanFunc based on strategy.

1166 -// Each strategy has different behavior:

1167 -// - "roots": Only root CIDs of pinned content

1168 -// - "pinned": All pinned content (roots + children)

1169 -// - "mfs": Only MFS content

1170 -// - "all": all blocks

1171 -func createKeyProvider(strategyFlag config.ProvideStrategy, in provStrategyIn) provider.KeyChanFunc {

1252 +// fpRate is the bloom filter target false-positive rate (1/N) used by

1253 +// +unique and +entities cycles. Ignored by other strategies.

1254 +func createKeyProvider(strategyFlag config.ProvideStrategy, fpRate uint, in provStrategyIn) provider.KeyChanFunc {

1255 + // +unique modifier: use bloom filter cross-DAG dedup

1256 + useUnique := strategyFlag&config.ProvideStrategyUnique != 0

1257 + if useUnique {

1258 + basePinned := strategyFlag&config.ProvideStrategyPinned != 0

1259 + baseMFS := strategyFlag&config.ProvideStrategyMFS != 0

1260 + ds := in.Repo.Datastore()

1261 +

1262 + // return a KeyChanFunc that creates a fresh bloom each cycle

1263 + return func(ctx context.Context) (<-chan cid.Cid, error) {

1264 + count := readLastUniqueCount(ds)

1265 + // size the bloom from the previous cycle's count (with growth

1266 + // margin for repo changes between cycles), falling back to

1267 + // DefaultBloomInitialCapacity on the very first cycle. The

1268 + // bloom chain auto-grows if the repo exceeds this estimate.

1269 + expectedItems := max(

1270 + uint64(walker.DefaultBloomInitialCapacity),

1271 + uint64(float64(count)*walker.BloomGrowthMargin),

1272 + )

1273 + // the tracker is shared across all sub-walks (MFS, recursive

1274 + // pins, direct pins) within a single reprovide cycle. it

1275 + // detects duplicate sub-DAG branches across recursive pins

1276 + // that share content (e.g. append-only datasets where each

1277 + // version differs by a small delta). when a CID is already

1278 + // in the bloom, its entire subtree is skipped, reducing

1279 + // traversal from O(pins * total_blocks) to O(unique_blocks).

1280 + tracker, err := walker.NewBloomTracker(uint(expectedItems), fpRate)

1281 + if err != nil {

1282 + return nil, fmt.Errorf("bloom tracker: %w", err)

1283 + }

1284 +

1285 + useEntities := strategyFlag&config.ProvideStrategyEntities != 0

1286 +

1287 + // select provider functions based on +entities modifier:

1288 + // +entities uses WalkEntityRoots (skips file chunks),

1289 + // +unique without +entities uses WalkDAG (all blocks).

1290 + makePinProv := dspinner.NewUniquePinnedProvider

1291 + makeMFSProv := uniqueMFSProvider

1292 + if useEntities {

1293 + makePinProv = dspinner.NewPinnedEntityRootsProvider

1294 + makeMFSProv = mfsEntityRootsProvider

1295 + }

1296 +

1297 + var inner provider.KeyChanFunc

1298 + switch {

1299 + case basePinned && baseMFS:

1300 + // MFS first: walk MFS (locality-filtered), then pinned.

1301 + // NewConcatProvider (not NewPrioritizedProvider) because

1302 + // the shared bloom tracker already guarantees each CID

1303 + // is emitted at most once -- no need for a second dedup

1304 + // layer. NewBufferedProvider decouples the pinned

1305 + // provider so the pinner lock is released promptly.

1306 + inner = provider.NewConcatProvider(

1307 + makeMFSProv(in.MFSRoot, in.Blockstore, tracker),

1308 + provider.NewBufferedProvider(

1309 + makePinProv(in.Pinner, in.Blockstore, tracker)),

1310 + )

1311 + case basePinned:

1312 + inner = provider.NewBufferedProvider(

1313 + makePinProv(in.Pinner, in.Blockstore, tracker))

1314 + case baseMFS:

1315 + inner = makeMFSProv(in.MFSRoot, in.Blockstore, tracker)

1316 + default:

1317 + return nil, fmt.Errorf("provider: +unique requires pinned and/or mfs")

1318 + }

1319 +

1320 + // wrap inner channel to persist bloom count on successful close

1321 + innerCh, err := inner(ctx)

1322 + if err != nil {

1323 + return nil, err

1324 + }

1325 +

1326 + ch := make(chan cid.Cid)

1327 + go func() {

1328 + defer func() {

1329 + if ctx.Err() == nil {

1330 + persistUniqueCount(ds, tracker.Count())

1331 + }

1332 + logger.Infow("unique reprovide cycle finished",

1333 + "providedCIDs", tracker.Count(),

1334 + "skippedBranches", tracker.Deduplicated())

1335 + close(ch)

1336 + }()

1337 + for c := range innerCh {

1338 + select {

1339 + case ch <- c:

1340 + case <-ctx.Done():

1341 + return

1342 + }

1343 + }

1344 + }()

1345 +

1346 + logger.Infow("unique reprovide cycle started",

1347 + "expectedItems", expectedItems,

1348 + "previousCount", count,

1349 + )

1350 + return ch, nil

1351 + }

1352 + }

1353 +

1354 + // non-unique strategies (unchanged)

1355 switch strategyFlag {

1356 case config.ProvideStrategyRoots:

1357 return provider.NewBufferedProvider(dspinner.NewPinnedProvider(true, in.Pinner, in.OfflineIPLDFetcher))

@@ -1239,12 +1422,12 @@ func handleStrategyChange(strategy string, provider DHTProvider, ds datastore.Da

1422 }

1423 }

1424

1242 -func setReproviderKeyProvider(strategy string) func(in provStrategyIn) provStrategyOut {

1243 - strategyFlag := config.ParseProvideStrategy(strategy)

1425 +func setReproviderKeyProvider(strategy string, fpRate uint) func(in provStrategyIn) provStrategyOut {

1426 + strategyFlag := config.MustParseProvideStrategy(strategy)

1427

1428 return func(in provStrategyIn) provStrategyOut {

1429 // Create the appropriate key provider based on strategy

1247 - kcf := createKeyProvider(strategyFlag, in)

1430 + kcf := createKeyProvider(strategyFlag, fpRate, in)

1431 return provStrategyOut{

1432 ProvidingStrategy: strategyFlag,

1433 ProvidingKeyChanFunc: kcf,

core/node/provider_test.go new

+91

@@ -0,0 +1,91 @@

1 +package node

2 +

3 +import (

4 + "context"

5 + "math"

6 + "testing"

7 +

8 + "github.com/ipfs/go-datastore"

9 + "github.com/stretchr/testify/assert"

10 + "github.com/stretchr/testify/require"

11 +)

12 +

13 +// newTestDatastore returns a fresh in-memory datastore for unique-count

14 +// persistence tests. Tests are single-goroutine so no sync wrapper is

15 +// needed.

16 +func newTestDatastore() datastore.Datastore {

17 + return datastore.NewMapDatastore()

18 +}

19 +

20 +func TestReadLastUniqueCount_emptyReturnsZero(t *testing.T) {

21 + ds := newTestDatastore()

22 +

23 + // A fresh datastore has no persisted count. The reader treats this

24 + // as "no previous cycle data available" and returns 0, which the

25 + // caller falls back to DefaultBloomInitialCapacity for.

26 + got := readLastUniqueCount(ds)

27 + assert.Equal(t, uint64(0), got)

28 +}

29 +

30 +func TestPersistAndReadUniqueCount_roundTrip(t *testing.T) {

31 + tests := []struct {

32 + name string

33 + count uint64

34 + }{

35 + {"zero", 0},

36 + {"one", 1},

37 + {"small", 1_000},

38 + {"million", 1_000_000},

39 + {"billion", 1_000_000_000},

40 + {"max uint64", math.MaxUint64},

41 + }

42 +

43 + for _, tt := range tests {

44 + t.Run(tt.name, func(t *testing.T) {

45 + ds := newTestDatastore()

46 + persistUniqueCount(ds, tt.count)

47 + got := readLastUniqueCount(ds)

48 + assert.Equal(t, tt.count, got)

49 + })

50 + }

51 +}

52 +

53 +func TestPersistUniqueCount_overwriteReplacesPreviousValue(t *testing.T) {

54 + ds := newTestDatastore()

55 +

56 + // Each reprovide cycle persists a new count, overwriting the

57 + // previous one. The reader must return the most recent value.

58 + persistUniqueCount(ds, 1_000)

59 + persistUniqueCount(ds, 2_000_000)

60 + persistUniqueCount(ds, 42)

61 +

62 + got := readLastUniqueCount(ds)

63 + assert.Equal(t, uint64(42), got)

64 +}

65 +

66 +func TestReadLastUniqueCount_corruptLengthReturnsZero(t *testing.T) {

67 + tests := []struct {

68 + name string

69 + raw []byte

70 + }{

71 + {"empty bytes", []byte{}},

72 + {"too short (4 bytes)", []byte{0x01, 0x02, 0x03, 0x04}},

73 + {"too long (16 bytes)", make([]byte, 16)},

74 + {"single byte", []byte{0xFF}},

75 + }

76 +

77 + for _, tt := range tests {

78 + t.Run(tt.name, func(t *testing.T) {

79 + ds := newTestDatastore()

80 + // Write malformed bytes directly under the persistence key

81 + // to simulate a corrupt or truncated entry.

82 + err := ds.Put(context.Background(), datastore.NewKey(reprovideLastUniqueCountKey), tt.raw)

83 + require.NoError(t, err)

84 +

85 + // The reader rejects anything that is not exactly 8 bytes

86 + // and falls back to 0 instead of panicking on a short read.

87 + got := readLastUniqueCount(ds)

88 + assert.Equal(t, uint64(0), got)

89 + })

90 + }

91 +}

core/node/storage.go

+1 -1

@@ -41,7 +41,7 @@ func BaseBlockstoreCtor(

41 // Important: Provide calls from blockstore are intentionally BLOCKING.

42 // The Provider implementation (not the blockstore) should handle concurrency/queuing.

43 // This avoids spawning unbounded goroutines for concurrent block additions.

44 - strategyFlag := config.ParseProvideStrategy(providingStrategy)

44 + strategyFlag := config.MustParseProvideStrategy(providingStrategy)

45 if strategyFlag&config.ProvideStrategyAll != 0 {

46 opts = append(opts, blockstore.Provider(prov))

47 }

docs/changelogs/v0.41.md

+40

@@ -14,6 +14,10 @@ This release was brought to you by the [Shipyard](https://ipshipyard.com/) team.

14 - [✨ New `ipfs cid inspect` command](#-new-ipfs-cid-inspect-command)

15 - [🖥️ WebUI Improvements](#-webui-improvements)

16 - [🔧 Correct provider addresses for custom HTTP routing](#-correct-provider-addresses-for-custom-http-routing)

17 + - [🔀 `Provide.Strategy` modifiers: `+unique` and `+entities`](#-providestrategy-modifiers-unique-and-entities)

18 + - [📌 `pin add` and `pin update` now fast-provide root CID](#-pin-add-and-pin-update-now-fast-provide-root-cid)

19 + - [🌳 New `--fast-provide-dag` flag for fine-tuned provide control](#-new---fast-provide-dag-flag-for-fine-tuned-provide-control)

20 + - [🛡️ Hardened `Provide.Strategy` parsing](#-hardened-providestrategy-parsing)

21 - [🛡️ `ipfs object patch` validates UnixFS node types](#-ipfs-object-patch-validates-unixfs-node-types)

22 - [🔗 MFS: fixed CidBuilder preservation](#-mfs-fixed-cidbuilder-preservation)

23 - [📂 FUSE Mount Improvements](#-fuse-mount-improvements)

@@ -80,6 +84,42 @@ Peer locations load faster thanks to UX optimizations in the underlying ipfs-geo

85 Nodes using custom routing (`Routing.Type=custom`) with [IPIP-526](https://github.com/ipfs/specs/pull/526) could end up publishing unresolved `0.0.0.0` addresses in provider records. Addresses are now resolved at provide-time, and when AutoNAT V2 has confirmed publicly reachable addresses, those are preferred automatically. See [#11213](https://github.com/ipfs/kubo/issues/11213).

87 +#### 🔀 `Provide.Strategy` modifiers: `+unique` and `+entities`

88 +

89 +Experimental opt-in optimizations for content providers with large repositories where multiple recursive pins share most of their DAG structure (e.g. append-only datasets, versioned archives like dist.ipfs.tech).

90 +

91 +- `+unique`: bloom filter dedup across recursive pins. Shared subtrees are traversed only once per reprovide cycle instead of once per pin, cutting I/O from O(pins * blocks) to O(unique blocks) at ~4 bytes/CID.

92 +- `+entities`: announces only entity roots (files, directories, HAMT shards), skipping internal file chunks. Drastically fewer DHT provider records while keeping all content discoverable by file/directory CID. Implies `+unique`.

93 +

94 +Example: `Provide.Strategy = "pinned+mfs+entities"`

95 +

96 +The default `Provide.Strategy=all` is unchanged. See [`Provide.Strategy`](https://github.com/ipfs/kubo/blob/master/docs/config.md#providestrategy) for configuration details and caveats.

97 +

98 +The bloom filter precision is tunable via [`Provide.BloomFPRate`](https://github.com/ipfs/kubo/blob/master/docs/config.md#providebloomfprate) (default ~1 false positive per 4.75M lookups, ~4 bytes per CID).

99 +

100 +#### 📌 `pin add` and `pin update` now fast-provide root CID

101 +

102 +`ipfs pin add` and `ipfs pin update` announce the pinned root CID to the routing system immediately after pinning, same as `ipfs add` and `ipfs dag import`. This matters for selective strategies like `pinned+mfs`, where previously the root CID was not announced until the next reprovide cycle (see [`Provide.DHT.Interval`](https://github.com/ipfs/kubo/blob/master/docs/config.md#providedhtinterval)). With the default `Provide.Strategy=all`, the blockstore already provides every block on write, so this is a no-op.

103 +

104 +Both commands now accept `--fast-provide-root`, `--fast-provide-dag`, and `--fast-provide-wait` flags, matching `ipfs add` and `ipfs dag import`. See [`Import`](https://github.com/ipfs/kubo/blob/master/docs/config.md#import) for defaults and configuration.

105 +

106 +#### 🌳 New `--fast-provide-dag` flag for fine-tuned provide control

107 +

108 +Users with a custom [`Provide.Strategy`](https://github.com/ipfs/kubo/blob/master/docs/config.md#providestrategy) (e.g. `pinned`, `pinned+mfs+entities`) now have finer control over which CIDs are announced immediately on `ipfs add`, `ipfs dag import`, `ipfs pin add`, and `ipfs pin update`.

109 +

110 +By default, only the root CID is provided right away (`--fast-provide-root=true`). Child blocks are deferred until the next [reprovide cycle](https://github.com/ipfs/kubo/blob/master/docs/config.md#providedhtinterval). This keeps bulk imports fast and avoids overwhelming online nodes with provide traffic.

111 +

112 +Pass `--fast-provide-dag=true` (or set [`Import.FastProvideDAG`](https://github.com/ipfs/kubo/blob/master/docs/config.md#importfastprovidedag)) to provide the full DAG immediately during add, using the active `Provide.Strategy` to determine scope.

113 +

114 +`Provide.Strategy=all` (default) is unaffected. It provides every block at the blockstore level regardless of this flag.

115 +

116 +> [!NOTE]

117 +> **Faster default imports for `Provide.Strategy=pinned` and `pinned+mfs` users.** Previously, `ipfs add --pin` eagerly announced every block of newly added content as it was written, through an internal DAG service wrapper. This release consolidates add-time providing through the new `--fast-provide-dag` code path, which defaults to `false`. The out-of-the-box result is faster bulk imports and less provide traffic during add: only the root CID is announced immediately (via [`Import.FastProvideRoot`](https://github.com/ipfs/kubo/blob/master/docs/config.md#importfastprovideroot)), and child blocks are picked up by the next reprovide cycle (see [`Provide.DHT.Interval`](https://github.com/ipfs/kubo/blob/master/docs/config.md#providedhtinterval), default 22h). To restore the previous eager-provide behavior on `ipfs add`, set [`Import.FastProvideDAG=true`](https://github.com/ipfs/kubo/blob/master/docs/config.md#importfastprovidedag) (or pass `--fast-provide-dag=true` per command); the walker honors the active `Provide.Strategy`. `Provide.Strategy=all` (the default) is unaffected.

118 +

119 +#### 🛡️ Hardened `Provide.Strategy` parsing

120 +

121 +Unknown strategy tokens (e.g. typo `"uniuqe"`), malformed delimiters (`"pinned+"`), and invalid combinations (`"all+pinned"`) now produce a clear error at startup instead of being silently ignored.

122 +

123 #### 🛡️ `ipfs object patch` validates UnixFS node types

124

125 As part of the ongoing deprecation of the legacy `ipfs object` API (which

docs/config.md

+111 -18

@@ -144,6 +144,7 @@ config file at runtime.

144 - [`Provide.DHT.MaxProvideConnsPerWorker`](#providedhtmaxprovideconnsperworker)

145 - [`Provide.DHT.KeystoreBatchSize`](#providedhtkeystorebatchsize)

146 - [`Provide.DHT.OfflineDelay`](#providedhtofflinedelay)

147 + - [`Provide.BloomFPRate`](#providebloomfprate)

148 - [`Provider`](#provider)

149 - [`Provider.Enabled`](#providerenabled)

150 - [`Provider.Strategy`](#providerstrategy)

@@ -238,6 +239,7 @@ config file at runtime.

239 - [`Import.UnixFSChunker`](#importunixfschunker)

240 - [`Import.HashFunction`](#importhashfunction)

241 - [`Import.FastProvideRoot`](#importfastprovideroot)

242 + - [`Import.FastProvideDAG`](#importfastprovidedag)

243 - [`Import.FastProvideWait`](#importfastprovidewait)

244 - [`Import.BatchMaxNodes`](#importbatchmaxnodes)

245 - [`Import.BatchMaxSize`](#importbatchmaxsize)

@@ -2090,31 +2092,74 @@ Type: `flag`

2092

2093 ### `Provide.Strategy`

2094

2093 -Tells the provide system what should be announced. Valid strategies are:

2095 +Controls which CIDs are announced to the content routing system. Valid strategies are:

2096

2097 - `"all"` - announce all CIDs of stored blocks

2098 - `"pinned"` - only announce recursively pinned CIDs (`ipfs pin add -r`, both roots and child blocks)

2099 - Order: root blocks of direct and recursive pins are announced first, then the child blocks of recursive pins

2098 -- `"roots"` - only announce the root block of explicitly pinned CIDs (`ipfs pin add`)

2099 - - **⚠️ BE CAREFUL:** node with `roots` strategy will not announce child blocks.

2100 +- `"roots"` - only announce the top-level root CID of explicitly pinned DAGs (`ipfs pin add`)

2101 + - **⚠️ BE CAREFUL:** a node with `roots` strategy will not announce child blocks.

2102 It makes sense only for use cases where the entire DAG is fetched in full,

2103 and a graceful resume does not have to be guaranteed: the lack of child

2104 announcements means an interrupted retrieval won't be able to find

2105 providers for the missing block in the middle of a file, unless the peer

2106 happens to already be connected to a provider and asks for child CID over

2105 - bitswap.

2107 + bitswap. Does not traverse the DAG to discover sub-entity roots

2108 + (files within directories, HAMT shards, etc.). If you want that, use

2109 + `"pinned+entities"` instead.

2110 - `"mfs"` - announce only the local CIDs that are part of the MFS (`ipfs files`)

2111 - Note: MFS is lazy-loaded. Only the MFS blocks present in local datastore are announced.

2112 - `"pinned+mfs"` - a combination of the `pinned` and `mfs` strategies.

2109 - - **ℹ️ NOTE:** This is the suggested strategy for users who run without GC and don't want to provide everything in cache.

2113 - Order: first `pinned` and then the locally available part of `mfs`.

2114

2112 -**Strategy changes automatically clear the provide queue.** When you change `Provide.Strategy` and restart Kubo, the provide queue is automatically cleared to ensure only content matching your new strategy is announced. You can also manually clear the queue using `ipfs provide clear`.

2115 +#### Strategy modifiers: `+unique` and `+entities`

2116 +

2117 +The `+unique` and `+entities` modifiers can be appended to `pinned`, `mfs`, or `pinned+mfs` strategies to optimize the reprovide cycle. They are incompatible with `"all"` and `"roots"`.

2118 +

2119 +- **`+unique`** -- uses a bloom filter to deduplicate CIDs across recursive

2120 + pins that share sub-DAGs. Without this, a node with 1000 pins that share 99%

2121 + of their content re-traverses the shared blocks for each pin. With `+unique`,

2122 + shared subtrees are detected and skipped, reducing traversal from

2123 + O(pins * total_blocks) to O(unique_blocks). This also significantly reduces

2124 + the amount of CIDs sent to the routing system when similar datasets are

2125 + pinned multiple times.

2126 +- **`+entities`** -- announces only entity roots (file roots, directory roots,

2127 + HAMT shard nodes) instead of every block. Internal file chunks are not

2128 + announced. This significantly reduces the number of provider records for

2129 + repositories with large files while keeping all files and directories

2130 + discoverable. Implies `+unique`. Non-UnixFS content (e.g. dag-cbor) is

2131 + still fully announced.

2132 + - **⚠️ BE CAREFUL:** since internal file chunks are not announced, resuming

2133 + an interrupted download from a specific byte offset or requesting a byte

2134 + range may not work unless the client is smart enough to find providers

2135 + for the entity root CID instead of the chunk CID. This is a work in

2136 + progress; see [kubo#10251](https://github.com/ipfs/kubo/issues/10251).

2137 +

2138 +**Suggested configurations:**

2139 +

2140 +- `"pinned+mfs+unique"` -- safe default for nodes with GC enabled, or desktop

2141 + users who don't want to announce all blocks cached in the local repository.

2142 + Handles pins of similar DAGs efficiently (e.g. versioned datasets where pins

2143 + are added and removed over time).

2144 +- `"pinned+mfs+entities"` -- same as above, but also skips internal file chunks

2145 + for even fewer provider records. Use when the `+entities` trade-off (no

2146 + chunk-level discoverability) is acceptable.

2147 +

2148 +#### Memory during reprovide

2149 +

2150 +Reproviding larger pinsets using the `mfs`, `pinned`, `pinned+mfs` or `roots` strategies requires additional memory, with an estimated ~1 GiB of RAM per 20 million CIDs. This is due to the use of a buffered provider, which loads all CIDs into memory to avoid holding a lock on the entire pinset during the reprovide cycle.

2151 +

2152 +With `+unique` or `+entities`, a bloom filter replaces the in-memory CID set, significantly reducing memory usage:

2153

2114 -**Memory requirements:**

2154 +- 2M CIDs: ~150 MB (default) vs ~8 MB (with `+unique` bloom filter)

2155 +- 10M CIDs: ~750 MB (default) vs ~42 MB (with `+unique` bloom filter)

2156 +- 100M CIDs: ~7.5 GB (default) vs ~713 MB (with `+unique` bloom filter)

2157

2116 -- Reproviding larger pinsets using the `mfs`, `pinned`, `pinned+mfs` or `roots` strategies requires additional memory, with an estimated ~1 GiB of RAM per 20 million CIDs for reproviding to the Amino DHT.

2117 -- This is due to the use of a buffered provider, which loads all CIDs into memory to avoid holding a lock on the entire pinset during the reprovide cycle.

2158 +The bloom auto-scales: the first cycle starts small and grows as needed; subsequent cycles size correctly from the previous cycle's count.

2159 +

2160 +#### Notes

2161 +

2162 +**Strategy changes automatically clear the provide queue.** When you change `Provide.Strategy` and restart Kubo, the provide queue is automatically cleared to ensure only content matching your new strategy is announced. You can also manually clear the queue using `ipfs provide clear`.

2163

2164 Default: `"all"`

2165

@@ -2472,6 +2517,42 @@ Default: `2h`

2517

2518 Type: `optionalDuration`

2519

2520 +### `Provide.BloomFPRate`

2521 +

2522 +Target false positive rate for the bloom filter used by the [`+unique` and

2523 +`+entities` strategy modifiers](#strategy-modifiers-unique-and-entities) and

2524 +the matching `--fast-provide-dag` walk. Expressed as `1/N` (one false positive

2525 +per `N` lookups), so a higher value means a lower FP rate but more memory per

2526 +CID. Has no effect when `Provide.Strategy` does not include `+unique` or

2527 +`+entities`.

2528 +

2529 +The bloom filter sizes itself from the previous reprovide cycle's CID count

2530 +and the configured FP rate. The auto-scaling described in

2531 +[Memory during reprovide](#memory-during-reprovide) is unaffected; this

2532 +setting only changes the bits-per-CID ratio of each bloom in the chain.

2533 +

2534 +Memory tradeoff (approximate, before `ipfs/bbloom`'s power-of-two rounding):

2535 +

2536 +| `Provide.BloomFPRate` | Approx. FP rate | Bytes per CID |

2537 +|-----------------------|-----------------|---------------|

2538 +| `1000000` | 1 in 1M | ~3 |

2539 +| (default) | ~1 in 4.75M | ~4 |

2540 +| `10000000` | 1 in 10M | ~5 |

2541 +| `100000000` | 1 in 100M | ~6 |

2542 +

2543 +A false positive causes the walker to skip a CID it has already been told

2544 +about; the skipped CID is provided in the next reprovide cycle (see

2545 +[`Provide.DHT.Interval`](#providedhtinterval)). At the default rate, fewer

2546 +than ~21 CIDs per 100M are skipped per cycle.

2547 +

2548 +The minimum accepted value is `1000000` (1 in 1M). Below that the bloom

2549 +filter becomes lossy enough to drop a meaningful fraction of CIDs from each

2550 +reprovide cycle.

2551 +

2552 +Default: `4750000` (~1 false positive per 4.75M lookups, cost at ~4 bytes per CID)

2553 +

2554 +Type: `optionalInteger`

2555 +

2556 ## `Provider`

2557

2558 ### `Provider.Enabled`

@@ -3791,29 +3872,41 @@ Type: `optionalString`

3872

3873 ### `Import.FastProvideRoot`

3874

3794 -Immediately provide root CIDs to the DHT in addition to the regular provide queue.

3795 -

3796 -This complements the sweep provider system: fast-provide handles the urgent case (root CIDs that users share and reference), while the sweep provider efficiently provides all blocks according to the `Provide.Strategy` over time. Together, they optimize for both immediate discoverability of newly imported content and efficient resource usage for complete DAG provides.

3875 +Immediately provide root CIDs to the routing system in addition to the regular provide queue.

3876

3798 -When disabled, only the sweep provider's queue is used.

3877 +This complements the reprovide system: fast-provide handles the urgent case (root CIDs that users share and reference), while the reprovide cycle provides all blocks according to the [`Provide.Strategy`](#providestrategy) over time.

3878

3800 -This setting applies to both `ipfs add` and `ipfs dag import` commands and can be overridden per-command with the `--fast-provide-root` flag.

3879 +When disabled, only the reprovide cycle handles content announcement.

3880

3802 -Ignored when DHT is not available for routing (e.g., `Routing.Type=none` or delegated-only configurations).

3881 +Applies to `ipfs add`, `ipfs dag import`, `ipfs pin add`, and `ipfs pin update`. Can be overridden per-command with the `--fast-provide-root` flag.

3882

3883 Default: `true`

3884

3885 Type: `flag`

3886

3887 +### `Import.FastProvideDAG`

3888 +

3889 +Walk and provide the full DAG immediately after content is added or pinned, using the active [`Provide.Strategy`](#providestrategy) to determine scope.

3890 +

3891 +When enabled with `+unique`, the DAG walk deduplicates via a bloom filter. When enabled with `+entities`, only entity roots (files, directories, HAMT shards) are provided.

3892 +

3893 +When disabled (default), only the root CID is provided immediately (via [`Import.FastProvideRoot`](#importfastprovideroot)) and child blocks are deferred to the reprovide cycle.

3894 +

3895 +Applies to `ipfs add`, `ipfs dag import`, `ipfs pin add`, and `ipfs pin update`. Can be overridden per-command with the `--fast-provide-dag` flag. Has no effect when `Provide.Strategy=all` (the blockstore already provides every block on write).

3896 +

3897 +Default: `false`

3898 +

3899 +Type: `flag`

3900 +

3901 ### `Import.FastProvideWait`

3902

3810 -Wait for the immediate root CID provide to complete before returning.

3903 +Wait for the immediate provide to complete before returning.

3904

3812 -When enabled, the command blocks until the provide completes, ensuring guaranteed discoverability before returning. When disabled (default), the provide happens asynchronously in the background without blocking the command.

3905 +When enabled, the command blocks until the provide completes, ensuring guaranteed discoverability before returning. When disabled (default), the provide happens asynchronously in the background without blocking the command. Applies to both [`Import.FastProvideRoot`](#importfastprovideroot) and [`Import.FastProvideDAG`](#importfastprovidedag).

3906

3907 Use this when you need certainty that content is discoverable before the command returns (e.g., sharing a link immediately after adding).

3908

3816 -This setting applies to both `ipfs add` and `ipfs dag import` commands and can be overridden per-command with the `--fast-provide-wait` flag.

3909 +Applies to `ipfs add`, `ipfs dag import`, `ipfs pin add`, and `ipfs pin update`. Can be overridden per-command with the `--fast-provide-wait` flag.

3910

3911 Ignored when DHT is not available for routing (e.g., `Routing.Type=none` or delegated-only configurations).

3912

docs/environment-variables.md

+22

@@ -267,6 +267,28 @@ Reducing it slows down connection ballooning but might affect performance negati

267

268 Default: [160](https://github.com/libp2p/go-libp2p/blob/master/p2p/net/swarm/swarm_dial.go#L91) (not set)

269

270 +## `TEST_DHT_STUB`

271 +

272 +Lifts WAN DHT filters so kubo can operate against DHT peers on

273 +loopback, enabling full end-to-end provide/findprovs/IPNS testing

274 +without public internet access. All DHT code paths are exercised:

275 +dial, protocol negotiation, message serialization, routing table

276 +management.

277 +

278 +Filters removed on the WAN DHT when this variable is set:

279 +

280 +- `AddressFilter`: accepts loopback addresses (default rejects non-public)

281 +- `QueryFilter`: accepts all peers (default rejects non-public)

282 +- `RoutingTableFilter`: accepts all peers (default rejects non-public)

283 +- `RoutingTablePeerDiversityFilter`: disabled (default caps same-IP peers to 3)

284 +

285 +In the CLI test harness, `h.BootstrapWithStubDHT(nodes)` spawns a

286 +mini-DHT on the loopback interface and sets this variable on each

287 +node automatically, allowing the loopback DHT to serve as a WAN

288 +replacement. Tests do not need to set this variable externally.

289 +

290 +Default: disabled (not set)

291 +

292 # Tracing

293

294 For tracing configuration, please check: https://github.com/ipfs/boxo/blob/main/docs/tracing.md

docs/examples/kubo-as-a-library/go.mod

+1 -1

@@ -118,7 +118,7 @@ require (

118 github.com/libp2p/go-doh-resolver v0.5.0 // indirect

119 github.com/libp2p/go-flow-metrics v0.3.0 // indirect

120 github.com/libp2p/go-libp2p-asn-util v0.4.1 // indirect

121 - github.com/libp2p/go-libp2p-kad-dht v0.39.0 // indirect

121 + github.com/libp2p/go-libp2p-kad-dht v0.39.1-0.20260326020727-bcbc21e9f633 // indirect

122 github.com/libp2p/go-libp2p-kbucket v0.8.0 // indirect

123 github.com/libp2p/go-libp2p-pubsub v0.15.0 // indirect

124 github.com/libp2p/go-libp2p-pubsub-router v0.6.0 // indirect

docs/examples/kubo-as-a-library/go.sum

+2 -2

@@ -494,8 +494,8 @@ github.com/libp2p/go-libp2p-asn-util v0.4.1 h1:xqL7++IKD9TBFMgnLPZR6/6iYhawHKHl9

494 github.com/libp2p/go-libp2p-asn-util v0.4.1/go.mod h1:d/NI6XZ9qxw67b4e+NgpQexCIiFYJjErASrYW4PFDN8=

495 github.com/libp2p/go-libp2p-core v0.2.4/go.mod h1:STh4fdfa5vDYr0/SzYYeqnt+E6KfEV5VxfIrm0bcI0g=

496 github.com/libp2p/go-libp2p-core v0.3.0/go.mod h1:ACp3DmS3/N64c2jDzcV429ukDpicbL6+TrrxANBjPGw=

497 -github.com/libp2p/go-libp2p-kad-dht v0.39.0 h1:mww38eBYiUvdsu+Xl/GLlBC0Aa8M+5HAwvafkFOygAM=

498 -github.com/libp2p/go-libp2p-kad-dht v0.39.0/go.mod h1:Po2JugFEkDq9Vig/JXtc153ntOi0q58o4j7IuITCOVs=

497 +github.com/libp2p/go-libp2p-kad-dht v0.39.1-0.20260326020727-bcbc21e9f633 h1:PcubpdBr1BBg39st+CqGp3EOX++DOBK6B/s07P31eMg=

498 +github.com/libp2p/go-libp2p-kad-dht v0.39.1-0.20260326020727-bcbc21e9f633/go.mod h1:Po2JugFEkDq9Vig/JXtc153ntOi0q58o4j7IuITCOVs=

499 github.com/libp2p/go-libp2p-kbucket v0.3.1/go.mod h1:oyjT5O7tS9CQurok++ERgc46YLwEpuGoFq9ubvoUOio=

500 github.com/libp2p/go-libp2p-kbucket v0.8.0 h1:QAK7RzKJpYe+EuSEATAaaHYMYLkPDGC18m9jxPLnU8s=

501 github.com/libp2p/go-libp2p-kbucket v0.8.0/go.mod h1:JMlxqcEyKwO6ox716eyC0hmiduSWZZl6JY93mGaaqc4=

test/cli/harness/dht_stub_peers.go new

+145

@@ -0,0 +1,145 @@

1 +package harness

2 +

3 +import (

4 + "context"

5 + "encoding/hex"

6 + "sync"

7 +

8 + "github.com/libp2p/go-libp2p"

9 + dht "github.com/libp2p/go-libp2p-kad-dht"

10 + "github.com/libp2p/go-libp2p-kad-dht/records"

11 + "github.com/libp2p/go-libp2p/core/host"

12 + "github.com/libp2p/go-libp2p/core/peer"

13 +)

14 +

15 +// stubPeerPool manages ephemeral in-process libp2p/DHT peers for

16 +// TEST_DHT_STUB mode.

17 +//

18 +// All peers share a single in-memory ProviderStore. This store is

19 +// NOT shared with the kubo daemons; it lives in the test process.

20 +// When a kubo daemon sends ADD_PROVIDER to any ephemeral peer, the

21 +// record is stored in this shared store. When another kubo daemon

22 +// queries GET_PROVIDERS from any peer, it finds the record because

23 +// all peers see the same store. The kubo daemons communicate with

24 +// the ephemeral peers via real DHT protocol messages over loopback

25 +// TCP.

26 +type stubPeerPool struct {

27 + hosts []host.Host

28 + dhts []*dht.IpfsDHT

29 + store *sharedMemStore

30 + cancel context.CancelFunc

31 +}

32 +

33 +// stubDHTPeerCount is the number of ephemeral DHT peers to create.

34 +// Matches amino.DefaultBucketSize (K=20 in Kademlia), ensuring

35 +// GetClosestPeers always finds enough peers for provide replication.

36 +const stubDHTPeerCount = 20

37 +

38 +// newStubPeerPool creates count ephemeral DHT peers on loopback and

39 +// mesh-connects them.

40 +func newStubPeerPool(count int) (*stubPeerPool, error) {

41 + ctx, cancel := context.WithCancel(context.Background())

42 +

43 + store := &sharedMemStore{data: make(map[string][]peer.AddrInfo)}

44 +

45 + hosts := make([]host.Host, 0, count)

46 + dhts := make([]*dht.IpfsDHT, 0, count)

47 +

48 + cleanup := func() {

49 + for _, d := range dhts {

50 + d.Close()

51 + }

52 + for _, h := range hosts {

53 + h.Close()

54 + }

55 + cancel()

56 + }

57 +

58 + for range count {

59 + h, err := libp2p.New(libp2p.ListenAddrStrings("/ip4/127.0.0.1/tcp/0"))

60 + if err != nil {

61 + cleanup()

62 + return nil, err

63 + }

64 + d, err := dht.New(ctx, h,

65 + dht.Mode(dht.ModeServer),

66 + dht.ProviderStore(store),

67 + dht.AddressFilter(nil),

68 + dht.DisableAutoRefresh(),

69 + dht.BootstrapPeers(),

70 + )

71 + if err != nil {

72 + h.Close()

73 + cleanup()

74 + return nil, err

75 + }

76 + hosts = append(hosts, h)

77 + dhts = append(dhts, d)

78 + }

79 +

80 + // Full-mesh connect so routing tables are populated.

81 + for i, h := range hosts {

82 + for j, other := range hosts {

83 + if i == j {

84 + continue

85 + }

86 + ai := peer.AddrInfo{ID: other.ID(), Addrs: other.Addrs()}

87 + if err := h.Connect(ctx, ai); err != nil {

88 + cleanup()

89 + return nil, err

90 + }

91 + }

92 + }

93 +

94 + return &stubPeerPool{

95 + hosts: hosts,

96 + dhts: dhts,

97 + store: store,

98 + cancel: cancel,

99 + }, nil

100 +}

101 +

102 +func (p *stubPeerPool) Close() {

103 + if p == nil {

104 + return

105 + }

106 + for _, d := range p.dhts {

107 + d.Close()

108 + }

109 + for _, h := range p.hosts {

110 + h.Close()

111 + }

112 + p.cancel()

113 +}

114 +

115 +// sharedMemStore implements records.ProviderStore with a shared

116 +// in-memory map. All ephemeral peers reference the same instance

117 +// so any peer can answer provider queries for any CID.

118 +type sharedMemStore struct {

119 + mu sync.RWMutex

120 + data map[string][]peer.AddrInfo

121 +}

122 +

123 +var _ records.ProviderStore = (*sharedMemStore)(nil)

124 +

125 +func (s *sharedMemStore) AddProvider(_ context.Context, key []byte, prov peer.AddrInfo) error {

126 + h := hex.EncodeToString(key)

127 + s.mu.Lock()

128 + defer s.mu.Unlock()

129 + for _, existing := range s.data[h] {

130 + if existing.ID == prov.ID {

131 + return nil

132 + }

133 + }

134 + s.data[h] = append(s.data[h], prov)

135 + return nil

136 +}

137 +

138 +func (s *sharedMemStore) GetProviders(_ context.Context, key []byte) ([]peer.AddrInfo, error) {

139 + h := hex.EncodeToString(key)

140 + s.mu.RLock()

141 + defer s.mu.RUnlock()

142 + return s.data[h], nil

143 +}

144 +

145 +func (s *sharedMemStore) Close() error { return nil }

test/cli/harness/harness.go

+36 -1

@@ -22,6 +22,7 @@ type Harness struct {

22 Runner *Runner

23 NodesRoot string

24 Nodes Nodes

25 + stubPeers *stubPeerPool // ephemeral DHT peers for TEST_DHT_STUB mode

26 }

28 // TODO: use zaptest.NewLogger(t) instead

@@ -73,6 +74,40 @@ func New(options ...func(h *Harness)) *Harness {

74 return h

75 }

77 +// BootstrapWithStubDHT configures each node to bootstrap from

78 +// ephemeral in-process DHT peers on loopback instead of the public

79 +// swarm. Call after Init() and before StartDaemon().

80 +//

81 +// Creates 20 ephemeral DHT peers lazily on the first call, shared

82 +// across all nodes in this harness. Sets TEST_DHT_STUB on each

83 +// node's environment so the daemon lifts WAN DHT filters to accept

84 +// loopback peers. Peers are shut down in Cleanup().

85 +//

86 +// The sweep provider needs >=20 DHT peers to estimate the network

87 +// size (prefix length). Without enough peers it stays offline and

88 +// never provides.

89 +func (h *Harness) BootstrapWithStubDHT(nodes Nodes) {

90 + if h.stubPeers == nil {

91 + pool, err := newStubPeerPool(stubDHTPeerCount)

92 + if err != nil {

93 + log.Panicf("creating stub peer pool: %s", err)

94 + }

95 + h.stubPeers = pool

96 + }

97 + var addrs []string

98 + for _, host := range h.stubPeers.hosts {

99 + for _, addr := range host.Addrs() {

100 + addrs = append(addrs, addr.String()+"/p2p/"+host.ID().String())

101 + }

102 + }

103 + for _, node := range nodes {

104 + node.SetIPFSConfig("Bootstrap", addrs)

105 + // Tell the daemon to lift WAN DHT filters so loopback

106 + // ephemeral peers enter the WAN routing table.

107 + node.Runner.Env["TEST_DHT_STUB"] = "1"

108 + }

109 +}

110 +

111 func osEnviron() map[string]string {

112 m := map[string]string{}

113 for _, entry := range os.Environ() {

@@ -183,7 +218,7 @@ func (h *Harness) Sh(expr string) *RunResult {

218 func (h *Harness) Cleanup() {

219 log.Debugf("cleaning up cluster")

220 h.Nodes.StopDaemons()

186 - // TODO: don't do this if test fails, not sure how?

221 + h.stubPeers.Close()

222 log.Debugf("removing harness dir")

223 err := os.RemoveAll(h.Dir)

224 if err != nil {

test/cli/migrations/migration_17_to_latest_test.go

+2 -2

@@ -213,8 +213,8 @@ func testInvalidStrategyMigration(t *testing.T) {

213 outputStr := string(output)

214 t.Logf("Daemon output with invalid strategy: %s", outputStr)

215

216 - // The error should mention unknown strategy

217 - require.Contains(t, outputStr, "unknown strategy", "Should report unknown strategy error")

216 + // The error should mention unknown strategy token

217 + require.Contains(t, outputStr, "unknown provide strategy token", "Should report unknown strategy error")

218 }

219

220 func testRepoProviderReproviderMigration(t *testing.T) {

test/cli/provider_test.go

+663 -137

@@ -8,6 +8,8 @@ import (

8 "net/http/httptest"

9 "os"

10 "path/filepath"

11 + "regexp"

12 + "strconv"

13 "strings"

14 "sync/atomic"

15 "testing"

@@ -21,27 +23,43 @@ import (

24 const (

25 timeStep = 20 * time.Millisecond

24 - timeout = time.Second

26 + timeout = 30 * time.Second

27 )

29 type cfgApplier func(*harness.Node)

29 -func runProviderSuite(t *testing.T, reprovide bool, apply cfgApplier) {

31 +// uniq appends a nanosecond timestamp to s, ensuring unique CIDs

32 +// across test runs and parallel subtests.

33 +func uniq(s string) string {

34 + return s + " " + strconv.FormatInt(time.Now().UnixNano(), 10)

35 +}

36 +

37 +// awaitReprovideFunc waits until at least minCIDs have been provided

38 +// and returns the total number of CIDs provided so far. The returned

39 +// count can be passed as minCIDs to a subsequent call to wait for the

40 +// next reprovide cycle.

41 +type awaitReprovideFunc func(t *testing.T, n *harness.Node, minCIDs int64) int64

42 +

43 +func runProviderSuite(t *testing.T, sweep bool, apply cfgApplier, awaitReprovide awaitReprovideFunc) {

44 t.Helper()

46 initNodes := func(t *testing.T, n int, fn func(n *harness.Node)) harness.Nodes {

33 - nodes := harness.NewT(t).NewNodes(n).Init()

47 + h := harness.NewT(t)

48 + nodes := h.NewNodes(n).Init()

49 nodes.ForEachPar(apply)

50 nodes.ForEachPar(fn)

51 + h.BootstrapWithStubDHT(nodes)

52 nodes = nodes.StartDaemons().Connect()

53 time.Sleep(500 * time.Millisecond) // wait for DHT clients to be bootstrapped

54 return nodes

55 }

57 initNodesWithoutStart := func(t *testing.T, n int, fn func(n *harness.Node)) harness.Nodes {

42 - nodes := harness.NewT(t).NewNodes(n).Init()

58 + h := harness.NewT(t)

59 + nodes := h.NewNodes(n).Init()

60 nodes.ForEachPar(apply)

61 nodes.ForEachPar(fn)

62 + h.BootstrapWithStubDHT(nodes)

63 return nodes

64 }

@@ -230,45 +248,47 @@ func runProviderSuite(t *testing.T, reprovide bool, apply cfgApplier) {

248 expectNoProviders(t, cid, nodes[1:]...)

249 })

250

233 - // It is a lesser evil - forces users to fix their config and have some sort of interval

234 - t.Run("Manual Reprovide trigger does not work when periodic reprovide is disabled", func(t *testing.T) {

235 - t.Parallel()

251 + // `routing reprovide` is only available with the legacy provider.

252 + // Sweep provider reprovides automatically on schedule.

253 + if !sweep {

254 + t.Run("Manual Reprovide trigger does not work when periodic reprovide is disabled", func(t *testing.T) {

255 + t.Parallel()

256

237 - nodes := initNodes(t, 2, func(n *harness.Node) {

238 - n.SetIPFSConfig("Provide.DHT.Interval", "0")

239 - })

240 - defer nodes.StopDaemons()

257 + nodes := initNodes(t, 2, func(n *harness.Node) {

258 + n.SetIPFSConfig("Provide.DHT.Interval", "0")

259 + })

260 + defer nodes.StopDaemons()

261

242 - cid := nodes[0].IPFSAddStr(time.Now().String())

262 + cid := nodes[0].IPFSAddStr(time.Now().String())

263

244 - expectNoProviders(t, cid, nodes[1:]...)

264 + expectNoProviders(t, cid, nodes[1:]...)

265

246 - res := nodes[0].RunIPFS("routing", "reprovide")

247 - assert.Contains(t, res.Stderr.Trimmed(), "invalid configuration: Provide.DHT.Interval is set to '0'")

248 - assert.Equal(t, 1, res.ExitCode())

266 + res := nodes[0].RunIPFS("routing", "reprovide")

267 + assert.Contains(t, res.Stderr.Trimmed(), "invalid configuration: Provide.DHT.Interval is set to '0'")

268 + assert.Equal(t, 1, res.ExitCode())

269

250 - expectNoProviders(t, cid, nodes[1:]...)

251 - })

270 + expectNoProviders(t, cid, nodes[1:]...)

271 + })

272

253 - // It is a lesser evil - forces users to fix their config and have some sort of interval

254 - t.Run("Manual Reprovide trigger does not work when Provide system is disabled", func(t *testing.T) {

255 - t.Parallel()

273 + t.Run("Manual Reprovide trigger does not work when Provide system is disabled", func(t *testing.T) {

274 + t.Parallel()

275

257 - nodes := initNodes(t, 2, func(n *harness.Node) {

258 - n.SetIPFSConfig("Provide.Enabled", false)

259 - })

260 - defer nodes.StopDaemons()

276 + nodes := initNodes(t, 2, func(n *harness.Node) {

277 + n.SetIPFSConfig("Provide.Enabled", false)

278 + })

279 + defer nodes.StopDaemons()

280

262 - cid := nodes[0].IPFSAddStr(time.Now().String())

281 + cid := nodes[0].IPFSAddStr(time.Now().String())

282

264 - expectNoProviders(t, cid, nodes[1:]...)

283 + expectNoProviders(t, cid, nodes[1:]...)

284

266 - res := nodes[0].RunIPFS("routing", "reprovide")

267 - assert.Contains(t, res.Stderr.Trimmed(), "invalid configuration: Provide.Enabled is set to 'false'")

268 - assert.Equal(t, 1, res.ExitCode())

285 + res := nodes[0].RunIPFS("routing", "reprovide")

286 + assert.Contains(t, res.Stderr.Trimmed(), "invalid configuration: Provide.Enabled is set to 'false'")

287 + assert.Equal(t, 1, res.ExitCode())

288

270 - expectNoProviders(t, cid, nodes[1:]...)

271 - })

289 + expectNoProviders(t, cid, nodes[1:]...)

290 + })

291 + }

292

293 t.Run("Provide with 'all' strategy", func(t *testing.T) {

294 t.Parallel()

@@ -277,9 +297,10 @@ func runProviderSuite(t *testing.T, reprovide bool, apply cfgApplier) {

297 n.SetIPFSConfig("Provide.Strategy", "all")

298 })

299 defer nodes.StopDaemons()

300 + publisher := nodes[0]

301

281 - cid := nodes[0].IPFSAddStr("all strategy")

282 - expectProviders(t, cid, nodes[0].PeerID().String(), nodes[1:]...)

302 + cid := publisher.IPFSAddStr(uniq("all strategy"))

303 + expectProviders(t, cid, publisher.PeerID().String(), nodes[1:]...)

304 })

305

306 t.Run("Provide with 'pinned' strategy", func(t *testing.T) {

@@ -289,14 +310,15 @@ func runProviderSuite(t *testing.T, reprovide bool, apply cfgApplier) {

310 n.SetIPFSConfig("Provide.Strategy", "pinned")

311 })

312 defer nodes.StopDaemons()

313 + publisher := nodes[0]

314

315 // Add a non-pinned CID (should not be provided)

294 - cid := nodes[0].IPFSAddStr("pinned strategy", "--pin=false")

316 + cid := publisher.IPFSAddStr(uniq("pinned strategy"), "--pin=false")

317 expectNoProviders(t, cid, nodes[1:]...)

318

319 // Pin the CID (should now be provided)

298 - nodes[0].IPFS("pin", "add", cid)

299 - expectProviders(t, cid, nodes[0].PeerID().String(), nodes[1:]...)

320 + publisher.IPFS("pin", "add", cid)

321 + expectProviders(t, cid, publisher.PeerID().String(), nodes[1:]...)

322 })

323

324 t.Run("Provide with 'pinned+mfs' strategy", func(t *testing.T) {

@@ -306,17 +328,148 @@ func runProviderSuite(t *testing.T, reprovide bool, apply cfgApplier) {

328 n.SetIPFSConfig("Provide.Strategy", "pinned+mfs")

329 })

330 defer nodes.StopDaemons()

331 + publisher := nodes[0]

332

310 - // Add a pinned CID (should be provided)

311 - cidPinned := nodes[0].IPFSAddStr("pinned content")

312 - cidUnpinned := nodes[0].IPFSAddStr("unpinned content", "--pin=false")

313 - cidMFS := nodes[0].IPFSAddStr("mfs content", "--pin=false")

314 - nodes[0].IPFS("files", "cp", "/ipfs/"+cidMFS, "/myfile")

333 + cidPinned := publisher.IPFSAddStr(uniq("pinned content"))

334 + cidUnpinned := publisher.IPFSAddStr(uniq("unpinned content"), "--pin=false")

335 + cidMFS := publisher.IPFSAddStr(uniq("mfs content"), "--pin=false")

336 + publisher.IPFS("files", "cp", "/ipfs/"+cidMFS, "/myfile")

337

316 - n0pid := nodes[0].PeerID().String()

317 - expectProviders(t, cidPinned, n0pid, nodes[1:]...)

338 + expectProviders(t, cidPinned, publisher.PeerID().String(), nodes[1:]...)

339 expectNoProviders(t, cidUnpinned, nodes[1:]...)

319 - expectProviders(t, cidMFS, n0pid, nodes[1:]...)

340 + expectProviders(t, cidMFS, publisher.PeerID().String(), nodes[1:]...)

341 + })

342 +

343 + // addLargeFileInSubdir adds a 2 MiB file inside /subdir/ in MFS and

344 + // returns the MFS root CID, the file root CID, and a chunk CID.

345 + // The file is large enough to be split into multiple blocks.

346 + // The resulting DAG: root-dir/subdir/largefile (2+ chunks).

347 + addLargeFileInSubdir := func(t *testing.T, publisher *harness.Node) (cidRoot, cidSubdir, cidFile, cidChunk string) {

348 + t.Helper()

349 + largeData := random.Bytes(2 * 1024 * 1024) // 2 MiB = 2 chunks at 1 MiB

350 +

351 + // Add file without pinning, then build directory structure in MFS

352 + cidFile = publisher.IPFSAdd(bytes.NewReader(largeData), "-Q", "--pin=false")

353 + publisher.IPFS("files", "mkdir", "-p", "/subdir")

354 + publisher.IPFS("files", "cp", "/ipfs/"+cidFile, "/subdir/largefile")

355 +

356 + // Get CIDs for the directory structure

357 + cidRoot = publisher.IPFS("files", "stat", "--hash", "/").Stdout.Trimmed()

358 + cidSubdir = publisher.IPFS("files", "stat", "--hash", "/subdir").Stdout.Trimmed()

359 +

360 + // Get a chunk CID from the file's DAG links

361 + dagOut := publisher.IPFS("dag", "get", cidFile)

362 + var dagNode struct {

363 + Links []struct {

364 + Hash map[string]string `json:"Hash"`

365 + } `json:"Links"`

366 + }

367 + require.NoError(t, json.Unmarshal(dagOut.Stdout.Bytes(), &dagNode))

368 + require.Greater(t, len(dagNode.Links), 1, "file should have multiple chunks")

369 + cidChunk = dagNode.Links[0].Hash["/"]

370 + require.NotEmpty(t, cidChunk)

371 +

372 + return cidRoot, cidSubdir, cidFile, cidChunk

373 + }

374 +

375 + // +unique and +entities tests verify which CIDs end up in the DHT

376 + // (strategy scope). Bloom filter deduplication correctness and

377 + // entity type detection are tested in boxo/dag/walker/*_test.go.

378 +

379 + t.Run("Provide with 'pinned+mfs+unique' strategy", func(t *testing.T) {

380 + t.Parallel()

381 +

382 + nodes := initNodes(t, 2, func(n *harness.Node) {

383 + n.SetIPFSConfig("Provide.Strategy", "pinned+mfs+unique")

384 + n.SetIPFSConfig("Import.UnixFSChunker", "size-1048576") // 1 MiB chunks

385 + })

386 + defer nodes.StopDaemons()

387 + publisher, peers := nodes[0], nodes[1:]

388 +

389 + // +unique provides all blocks in pinned DAGs (same scope as

390 + // pinned+mfs but with bloom filter dedup across pins).

391 + // Use --fast-provide-dag and --fast-provide-wait on pin add

392 + // so we can verify which blocks the strategy includes.

393 + cidRoot, cidSubdir, cidFile, cidChunk := addLargeFileInSubdir(t, publisher)

394 + publisher.IPFS("pin", "add", "--fast-provide-dag", "--fast-provide-wait", cidRoot)

395 + cidUnpinned := publisher.IPFSAddStr(uniq("unpinned content"), "--pin=false")

396 +

397 + pid := publisher.PeerID().String()

398 + // All blocks in the pinned DAG should be provided (including chunks)

399 + expectProviders(t, cidRoot, pid, peers...)

400 + expectProviders(t, cidSubdir, pid, peers...)

401 + expectProviders(t, cidFile, pid, peers...)

402 + expectProviders(t, cidChunk, pid, peers...)

403 + expectNoProviders(t, cidUnpinned, peers...)

404 + })

405 +

406 + t.Run("Provide with 'pinned+mfs+entities' strategy", func(t *testing.T) {

407 + t.Parallel()

408 +

409 + nodes := initNodes(t, 2, func(n *harness.Node) {

410 + n.SetIPFSConfig("Provide.Strategy", "pinned+mfs+entities")

411 + n.SetIPFSConfig("Import.UnixFSChunker", "size-1048576") // 1 MiB chunks

412 + })

413 + defer nodes.StopDaemons()

414 + publisher, peers := nodes[0], nodes[1:]

415 +

416 + // +entities provides only entity roots (files, directories,

417 + // HAMT shards) and skips internal file chunks.

418 + // Use --fast-provide-dag and --fast-provide-wait on pin add

419 + // so we can verify which blocks the strategy skips.

420 + cidRoot, cidSubdir, cidFile, cidChunk := addLargeFileInSubdir(t, publisher)

421 + publisher.IPFS("pin", "add", "--fast-provide-dag", "--fast-provide-wait", cidRoot)

422 +

423 + pid := publisher.PeerID().String()

424 + // Entity roots: directories and file root

425 + expectProviders(t, cidRoot, pid, peers...)

426 + expectProviders(t, cidSubdir, pid, peers...)

427 + expectProviders(t, cidFile, pid, peers...)

428 + // Internal chunk should NOT be provided (+entities skips chunks)

429 + expectNoProviders(t, cidChunk, peers...)

430 + })

431 +

432 + t.Run("ipfs add --fast-provide-dag honors +entities (no chunk providing)", func(t *testing.T) {

433 + t.Parallel()

434 +

435 + nodes := initNodes(t, 2, func(n *harness.Node) {

436 + n.SetIPFSConfig("Provide.Strategy", "pinned+entities")

437 + n.SetIPFSConfig("Import.UnixFSChunker", "size-1048576") // 1 MiB chunks

438 + })

439 + defer nodes.StopDaemons()

440 + publisher, peers := nodes[0], nodes[1:]

441 +

442 + // Regression test for the providingDagService double-providing

443 + // path. Before the fix, ipfs add --pin --fast-provide-dag wrapped

444 + // the DAGService with providingDagService, which announced every

445 + // block as it was written -- including chunks -- regardless of

446 + // the +entities modifier. The post-add ExecuteFastProvideDAG

447 + // walk then ran in parallel, so chunks ended up in the DHT

448 + // despite +entities saying they should be skipped.

449 + //

450 + // After the fix, ExecuteFastProvideDAG is the single mechanism

451 + // for --fast-provide-dag and respects the active strategy.

452 + largeData := random.Bytes(2 * 1024 * 1024) // 2 MiB = 2 chunks

453 + cidFile := publisher.IPFSAdd(bytes.NewReader(largeData),

454 + "--fast-provide-dag", "--fast-provide-wait")

455 +

456 + // Get a chunk CID from the file's DAG links

457 + dagOut := publisher.IPFS("dag", "get", cidFile)

458 + var dagNode struct {

459 + Links []struct {

460 + Hash map[string]string `json:"Hash"`

461 + } `json:"Links"`

462 + }

463 + require.NoError(t, json.Unmarshal(dagOut.Stdout.Bytes(), &dagNode))

464 + require.Greater(t, len(dagNode.Links), 1, "file should have multiple chunks")

465 + cidChunk := dagNode.Links[0].Hash["/"]

466 + require.NotEmpty(t, cidChunk)

467 +

468 + pid := publisher.PeerID().String()

469 + // File root (entity) should be provided

470 + expectProviders(t, cidFile, pid, peers...)

471 + // Chunk should NOT be provided (+entities skips chunks)

472 + expectNoProviders(t, cidChunk, peers...)

473 })

474

475 t.Run("Provide with 'roots' strategy", func(t *testing.T) {

@@ -326,13 +479,17 @@ func runProviderSuite(t *testing.T, reprovide bool, apply cfgApplier) {

479 n.SetIPFSConfig("Provide.Strategy", "roots")

480 })

481 defer nodes.StopDaemons()

482 + publisher := nodes[0]

483

330 - // Add a root CID (should be provided)

331 - cidRoot := nodes[0].IPFSAddStr("roots strategy", "-w", "-Q")

332 - // the same without wrapping should give us a child node.

333 - cidChild := nodes[0].IPFSAddStr("root strategy", "--pin=false")

484 + // Add with -w: the wrapper directory is the recursive pin root,

485 + // the file inside is a child block of that pin (not a root).

486 + // Use --only-hash first to learn the child CID without providing.

487 + data := random.Bytes(1000)

488 + cidChild := publisher.IPFSAdd(bytes.NewReader(data), "-Q", "--only-hash")

489 + cidRoot := publisher.IPFSAdd(bytes.NewReader(data), "-Q", "-w")

490

335 - expectProviders(t, cidRoot, nodes[0].PeerID().String(), nodes[1:]...)

491 + // 'roots' strategy provides only pin roots, not child blocks.

492 + expectProviders(t, cidRoot, publisher.PeerID().String(), nodes[1:]...)

493 expectNoProviders(t, cidChild, nodes[1:]...)

494 })

495

@@ -343,19 +500,66 @@ func runProviderSuite(t *testing.T, reprovide bool, apply cfgApplier) {

500 n.SetIPFSConfig("Provide.Strategy", "mfs")

501 })

502 defer nodes.StopDaemons()

503 + publisher := nodes[0]

504

347 - // Add a file to MFS (should be provided)

348 - data := random.Bytes(1000)

349 - cid := nodes[0].IPFSAdd(bytes.NewReader(data), "-Q")

505 + // 'mfs' only provides content in MFS. Pinned content outside

506 + // MFS should NOT be provided (mfs excludes pinned by default).

507 + cidPinned := publisher.IPFSAddStr(uniq("pinned but not mfs"))

508 + expectNoProviders(t, cidPinned, nodes[1:]...)

509

351 - // not yet in MFS

352 - expectNoProviders(t, cid, nodes[1:]...)

510 + // Add to MFS (should be provided)

511 + data := random.Bytes(1000)

512 + cidMFS := publisher.IPFSAdd(bytes.NewReader(data), "-Q", "--pin=false")

513 + publisher.IPFS("files", "cp", "/ipfs/"+cidMFS, "/myfile")

514 + expectProviders(t, cidMFS, publisher.PeerID().String(), nodes[1:]...)

515

354 - nodes[0].IPFS("files", "cp", "/ipfs/"+cid, "/myfile")

355 - expectProviders(t, cid, nodes[0].PeerID().String(), nodes[1:]...)

516 + // Pinned CID still not provided (mfs strategy ignores pins)

517 + expectNoProviders(t, cidPinned, nodes[1:]...)

518 })

519

358 - if reprovide {

520 + // Reprovide tests: add content offline, start daemon, wait for reprovide.

521 + //

522 + // Each test waits for TWO reprovide cycles to confirm the schedule

523 + // works repeatedly, not just on the initial bootstrap. The second

524 + // cycle also catches bugs where state isn't persisted across cycles.

525 + //

526 + // Legacy: `routing reprovide` blocks until the reprovide cycle finishes,

527 + // so we call it and check results immediately after.

528 + //

529 + // Sweep: no manual trigger exists. Instead, we set

530 + // Provide.DHT.Interval=30s on the importing node and poll

531 + // `provide stat` until the cycle completes.

532 +

533 + // verifyReprovide waits for two reprovide cycles and asserts which

534 + // CIDs are/aren't findable after each. minCIDs is the expected

535 + // number of provided CIDs per cycle.

536 + verifyReprovide := func(

537 + t *testing.T,

538 + publisher *harness.Node,

539 + queriers harness.Nodes,

540 + minCIDs int64,

541 + provided []string,

542 + notProvided []string,

543 + ) {

544 + t.Helper()

545 + pid := publisher.PeerID().String()

546 + check := func() {

547 + for _, c := range provided {

548 + expectProviders(t, c, pid, queriers...)

549 + }

550 + for _, c := range notProvided {

551 + expectNoProviders(t, c, queriers...)

552 + }

553 + }

554 +

555 + after1 := awaitReprovide(t, publisher, minCIDs)

556 + check()

557 + // Second cycle: confirms the schedule runs repeatedly.

558 + awaitReprovide(t, publisher, after1+minCIDs)

559 + check()

560 + }

561 +

562 + {

563

564 t.Run("Reprovides with 'all' strategy when strategy is '' (empty)", func(t *testing.T) {

565 t.Parallel()

@@ -363,16 +567,19 @@ func runProviderSuite(t *testing.T, reprovide bool, apply cfgApplier) {

567 nodes := initNodesWithoutStart(t, 2, func(n *harness.Node) {

568 n.SetIPFSConfig("Provide.Strategy", "")

569 })

570 + publisher := nodes[0]

571 + if sweep {

572 + publisher.SetIPFSConfig("Provide.DHT.Interval", "30s")

573 + }

574

367 - cid := nodes[0].IPFSAddStr(time.Now().String())

575 + cid := publisher.IPFSAddStr(time.Now().String())

576

577 nodes = nodes.StartDaemons().Connect()

578 defer nodes.StopDaemons()

371 - expectNoProviders(t, cid, nodes[1:]...)

372 -

373 - nodes[0].IPFS("routing", "reprovide")

579 + peers := nodes[1:]

580

375 - expectProviders(t, cid, nodes[0].PeerID().String(), nodes[1:]...)

581 + verifyReprovide(t, publisher, peers, 1, // 1 block added

582 + []string{cid}, nil)

583 })

584

585 t.Run("Reprovides with 'all' strategy", func(t *testing.T) {

@@ -381,16 +588,19 @@ func runProviderSuite(t *testing.T, reprovide bool, apply cfgApplier) {

588 nodes := initNodesWithoutStart(t, 2, func(n *harness.Node) {

589 n.SetIPFSConfig("Provide.Strategy", "all")

590 })

591 + publisher := nodes[0]

592 + if sweep {

593 + publisher.SetIPFSConfig("Provide.DHT.Interval", "30s")

594 + }

595

385 - cid := nodes[0].IPFSAddStr(time.Now().String())

596 + cid := publisher.IPFSAddStr(time.Now().String())

597

598 nodes = nodes.StartDaemons().Connect()

599 defer nodes.StopDaemons()

389 - expectNoProviders(t, cid, nodes[1:]...)

600 + peers := nodes[1:]

601

391 - nodes[0].IPFS("routing", "reprovide")

392 -

393 - expectProviders(t, cid, nodes[0].PeerID().String(), nodes[1:]...)

602 + verifyReprovide(t, publisher, peers, 1, // 1 block added

603 + []string{cid}, nil)

604 })

605

606 t.Run("Reprovides with 'pinned' strategy", func(t *testing.T) {

@@ -402,62 +612,54 @@ func runProviderSuite(t *testing.T, reprovide bool, apply cfgApplier) {

612 nodes := initNodesWithoutStart(t, 2, func(n *harness.Node) {

613 n.SetIPFSConfig("Provide.Strategy", "pinned")

614 })

615 + publisher := nodes[0]

616 + if sweep {

617 + publisher.SetIPFSConfig("Provide.DHT.Interval", "30s")

618 + }

619

406 - // Add a pin while offline so it cannot be provided

407 - cidBarDir := nodes[0].IPFSAdd(bytes.NewReader(bar), "-Q", "-w")

620 + // Add a pin while offline

621 + cidBarDir := publisher.IPFSAdd(bytes.NewReader(bar), "-Q", "-w")

622

623 nodes = nodes.StartDaemons().Connect()

624 defer nodes.StopDaemons()

625 + peers := nodes[1:]

626

412 - // Add content without pinning while daemon line

413 - cidFoo := nodes[0].IPFSAdd(bytes.NewReader(foo), "--pin=false")

414 - cidBar := nodes[0].IPFSAdd(bytes.NewReader(bar), "--pin=false")

415 -

416 - // Nothing should have been provided. The pin was offline, and

417 - // the others should not be provided per the strategy.

418 - expectNoProviders(t, cidFoo, nodes[1:]...)

419 - expectNoProviders(t, cidBar, nodes[1:]...)

420 - expectNoProviders(t, cidBarDir, nodes[1:]...)

627 + // Add content without pinning while daemon is online

628 + cidFoo := publisher.IPFSAdd(bytes.NewReader(foo), "--pin=false")

629 + cidBar := publisher.IPFSAdd(bytes.NewReader(bar), "--pin=false")

630

422 - nodes[0].IPFS("routing", "reprovide")

423 -

424 - // cidFoo is not pinned so should not be provided.

425 - expectNoProviders(t, cidFoo, nodes[1:]...)

426 - // cidBar gets provided by being a child from cidBarDir even though we added with pin=false.

427 - expectProviders(t, cidBar, nodes[0].PeerID().String(), nodes[1:]...)

428 - expectProviders(t, cidBarDir, nodes[0].PeerID().String(), nodes[1:]...)

631 + verifyReprovide(t, publisher, peers, 2, // cidBar + cidBarDir (bar is child of the wrapped dir pin)

632 + []string{cidBar, cidBarDir},

633 + []string{cidFoo}) // cidFoo not pinned

634 })

635

636 t.Run("Reprovides with 'roots' strategy", func(t *testing.T) {

637 t.Parallel()

638

434 - foo := random.Bytes(1000)

639 bar := random.Bytes(1000)

640

641 nodes := initNodesWithoutStart(t, 2, func(n *harness.Node) {

642 n.SetIPFSConfig("Provide.Strategy", "roots")

643 })

440 - n0pid := nodes[0].PeerID().String()

644 + publisher := nodes[0]

645 + if sweep {

646 + publisher.SetIPFSConfig("Provide.DHT.Interval", "30s")

647 + }

648

442 - // Add a pin. Only root should get pinned but not provided

443 - // because node not started

444 - cidBarDir := nodes[0].IPFSAdd(bytes.NewReader(bar), "-Q", "-w")

649 + // Compute the child CID without storing anything (safe

650 + // offline, daemon not started yet).

651 + cidChild := publisher.IPFSAdd(bytes.NewReader(bar), "-Q", "--only-hash")

652 + // Add with -w: pins the wrapper directory as root. The file

653 + // inside is a child block of that pin, not a root.

654 + cidRoot := publisher.IPFSAdd(bytes.NewReader(bar), "-Q", "-w")

655

656 nodes = nodes.StartDaemons().Connect()

657 defer nodes.StopDaemons()

658 + peers := nodes[1:]

659

449 - cidFoo := nodes[0].IPFSAdd(bytes.NewReader(foo))

450 - cidBar := nodes[0].IPFSAdd(bytes.NewReader(bar), "--pin=false")

451 -

452 - // cidFoo will get provided per the strategy but cidBar will not.

453 - expectProviders(t, cidFoo, n0pid, nodes[1:]...)

454 - expectNoProviders(t, cidBar, nodes[1:]...)

455 -

456 - nodes[0].IPFS("routing", "reprovide")

457 -

458 - expectProviders(t, cidFoo, n0pid, nodes[1:]...)

459 - expectNoProviders(t, cidBar, nodes[1:]...)

460 - expectProviders(t, cidBarDir, n0pid, nodes[1:]...)

660 + verifyReprovide(t, publisher, peers, 1, // cidRoot (only pin root)

661 + []string{cidRoot},

662 + []string{cidChild}) // child of pin, not a root

663 })

664

665 t.Run("Reprovides with 'mfs' strategy", func(t *testing.T) {

@@ -468,22 +670,24 @@ func runProviderSuite(t *testing.T, reprovide bool, apply cfgApplier) {

670 nodes := initNodesWithoutStart(t, 2, func(n *harness.Node) {

671 n.SetIPFSConfig("Provide.Strategy", "mfs")

672 })

471 - n0pid := nodes[0].PeerID().String()

673 + publisher := nodes[0]

674 + if sweep {

675 + publisher.SetIPFSConfig("Provide.DHT.Interval", "30s")

676 + }

677

473 - // add something and lets put it in MFS

474 - cidBar := nodes[0].IPFSAdd(bytes.NewReader(bar), "--pin=false", "-Q")

475 - nodes[0].IPFS("files", "cp", "/ipfs/"+cidBar, "/myfile")

678 + // Add to MFS (should be provided)

679 + cidMFS := publisher.IPFSAdd(bytes.NewReader(bar), "--pin=false", "-Q")

680 + publisher.IPFS("files", "cp", "/ipfs/"+cidMFS, "/myfile")

681 + // Pin something NOT in MFS (should NOT be provided)

682 + cidPinned := publisher.IPFSAddStr(uniq("pinned but not mfs"))

683

684 nodes = nodes.StartDaemons().Connect()

685 defer nodes.StopDaemons()

686 + peers := nodes[1:]

687

480 - // cidBar is in MFS but not provided

481 - expectNoProviders(t, cidBar, nodes[1:]...)

482 -

483 - nodes[0].IPFS("routing", "reprovide")

484 -

485 - // And now is provided

486 - expectProviders(t, cidBar, n0pid, nodes[1:]...)

688 + verifyReprovide(t, publisher, peers, 1, // cidMFS only

689 + []string{cidMFS},

690 + []string{cidPinned}) // mfs strategy ignores pinned content outside MFS

691 })

692

693 t.Run("Reprovides with 'pinned+mfs' strategy", func(t *testing.T) {

@@ -492,28 +696,79 @@ func runProviderSuite(t *testing.T, reprovide bool, apply cfgApplier) {

696 nodes := initNodesWithoutStart(t, 2, func(n *harness.Node) {

697 n.SetIPFSConfig("Provide.Strategy", "pinned+mfs")

698 })

495 - n0pid := nodes[0].PeerID().String()

699 + publisher := nodes[0]

700 + if sweep {

701 + publisher.SetIPFSConfig("Provide.DHT.Interval", "30s")

702 + }

703

704 // Add a pinned CID (should be provided)

498 - cidPinned := nodes[0].IPFSAddStr("pinned content", "--pin=true")

705 + cidPinned := publisher.IPFSAddStr(uniq("pinned content"), "--pin=true")

706 // Add a CID to MFS (should be provided)

500 - cidMFS := nodes[0].IPFSAddStr("mfs content")

501 - nodes[0].IPFS("files", "cp", "/ipfs/"+cidMFS, "/myfile")

707 + cidMFS := publisher.IPFSAddStr(uniq("mfs content"))

708 + publisher.IPFS("files", "cp", "/ipfs/"+cidMFS, "/myfile")

709 // Add a CID that is neither pinned nor in MFS (should not be provided)

503 - cidNeither := nodes[0].IPFSAddStr("neither content", "--pin=false")

710 + cidNeither := publisher.IPFSAddStr(uniq("neither content"), "--pin=false")

711 +

712 + nodes = nodes.StartDaemons().Connect()

713 + defer nodes.StopDaemons()

714 + peers := nodes[1:]

715 +

716 + verifyReprovide(t, publisher, peers, 2, // cidPinned + cidMFS

717 + []string{cidPinned, cidMFS},

718 + []string{cidNeither}) // neither pinned nor in MFS

719 + })

720 +

721 + t.Run("Reprovides with 'pinned+mfs+unique' strategy", func(t *testing.T) {

722 + t.Parallel()

723 +

724 + nodes := initNodesWithoutStart(t, 2, func(n *harness.Node) {

725 + n.SetIPFSConfig("Provide.Strategy", "pinned+mfs+unique")

726 + n.SetIPFSConfig("Import.UnixFSChunker", "size-1048576") // 1 MiB chunks

727 + })

728 + publisher := nodes[0]

729 + if sweep {

730 + publisher.SetIPFSConfig("Provide.DHT.Interval", "30s")

731 + }

732 +

733 + // Build a directory DAG with a multi-chunk file in MFS, then pin it.

734 + cidRoot, cidSubdir, cidFile, cidChunk := addLargeFileInSubdir(t, publisher)

735 + publisher.IPFS("pin", "add", cidRoot)

736 + cidUnpinned := publisher.IPFSAddStr(uniq("unpinned content"), "--pin=false")

737

738 nodes = nodes.StartDaemons().Connect()

739 defer nodes.StopDaemons()

740 + peers := nodes[1:]

741 +

742 + // +unique provides all blocks in pinned DAGs (same as pinned+mfs)

743 + verifyReprovide(t, publisher, peers, 4, // root + subdir + file + chunks

744 + []string{cidRoot, cidSubdir, cidFile, cidChunk},

745 + []string{cidUnpinned})

746 + })

747 +

748 + t.Run("Reprovides with 'pinned+mfs+entities' strategy", func(t *testing.T) {

749 + t.Parallel()

750 +

751 + nodes := initNodesWithoutStart(t, 2, func(n *harness.Node) {

752 + n.SetIPFSConfig("Provide.Strategy", "pinned+mfs+entities")

753 + n.SetIPFSConfig("Import.UnixFSChunker", "size-1048576") // 1 MiB chunks

754 + })

755 + publisher := nodes[0]

756 + if sweep {

757 + publisher.SetIPFSConfig("Provide.DHT.Interval", "30s")

758 + }

759

508 - // Trigger reprovide

509 - nodes[0].IPFS("routing", "reprovide")

760 + // Build a directory DAG with a multi-chunk file in MFS, then pin it.

761 + cidRoot, cidSubdir, cidFile, cidChunk := addLargeFileInSubdir(t, publisher)

762 + publisher.IPFS("pin", "add", cidRoot)

763

511 - // Check that pinned CID is provided

512 - expectProviders(t, cidPinned, n0pid, nodes[1:]...)

513 - // Check that MFS CID is provided

514 - expectProviders(t, cidMFS, n0pid, nodes[1:]...)

515 - // Check that neither CID is not provided

516 - expectNoProviders(t, cidNeither, nodes[1:]...)

764 + nodes = nodes.StartDaemons().Connect()

765 + defer nodes.StopDaemons()

766 + peers := nodes[1:]

767 +

768 + // Entity roots: directories and file root (not chunks)

769 + verifyReprovide(t, publisher, peers, 3, // root + subdir + file (not chunks)

770 + []string{cidRoot, cidSubdir, cidFile},

771 + []string{cidChunk}) // chunks skipped by +entities

772 })

773 }

774

@@ -714,12 +969,22 @@ type provideStatJSON struct {

969 Schedule struct {

970 NextReprovidePrefix string `json:"next_reprovide_prefix"`

971 } `json:"schedule"`

972 + Operations struct {

973 + Ongoing struct {

974 + KeyReprovides int `json:"key_reprovides"`

975 + } `json:"ongoing"`

976 + Past struct {

977 + KeysProvided int64 `json:"keys_provided"`

978 + } `json:"past"`

979 + } `json:"operations"`

980 + Queues struct {

981 + PendingKeyProvides int64 `json:"pending_key_provides"`

982 + } `json:"queues"`

983 } `json:"Sweep"`

984 }

985

986 // parseProvideStatJSON extracts timing and schedule information from

987 // the JSON output of 'ipfs provide stat --enc=json'.

722 -// Note: prefix is unused in current tests but kept for potential future use.

988 func parseProvideStatJSON(output string) (offset time.Duration, prefix string, err error) {

989 var stat provideStatJSON

990 if err := json.Unmarshal([]byte(output), &stat); err != nil {

@@ -730,43 +995,304 @@ func parseProvideStatJSON(output string) (offset time.Duration, prefix string, e

995 return offset, prefix, nil

996 }

997

998 +// waitForSweepReprovide polls `provide stat --enc=json` until the

999 +// sweep provider has provided at least minCIDs and no work is pending.

1000 +// Pass 0 for minCIDs to just wait for any provide activity to finish.

1001 +// Returns the total CIDs provided so far (for use as minCIDs in a

1002 +// subsequent call to wait for the next cycle).

1003 +// The importing node must have a short Provide.DHT.Interval so the

1004 +// reprovide cycle completes within the timeout.

1005 +func waitForSweepReprovide(t *testing.T, n *harness.Node, timeout time.Duration, minCIDs int64) int64 {

1006 + t.Helper()

1007 + if minCIDs == 0 {

1008 + minCIDs = 1

1009 + }

1010 + deadline := time.Now().Add(timeout)

1011 + for time.Now().Before(deadline) {

1012 + res := n.RunIPFS("provide", "stat", "--enc=json")

1013 + if res.ExitCode() == 0 {

1014 + var stat provideStatJSON

1015 + if err := json.Unmarshal(res.Stdout.Bytes(), &stat); err == nil {

1016 + s := stat.Sweep

1017 + if s.Operations.Past.KeysProvided >= minCIDs &&

1018 + s.Queues.PendingKeyProvides == 0 &&

1019 + s.Operations.Ongoing.KeyReprovides == 0 {

1020 + return s.Operations.Past.KeysProvided

1021 + }

1022 + }

1023 + }

1024 + time.Sleep(500 * time.Millisecond)

1025 + }

1026 + t.Fatalf("sweep reprovide: expected at least %d CIDs provided within %s", minCIDs, timeout)

1027 + return 0

1028 +}

1029 +

1030 func TestProvider(t *testing.T) {

1031 t.Parallel()

1032

1033 variants := []struct {

737 - name string

738 - reprovide bool

739 - apply cfgApplier

1034 + name string

1035 + sweep bool

1036 + apply cfgApplier

1037 + awaitReprovide awaitReprovideFunc

1038 }{

1039 {

742 - name: "LegacyProvider",

743 - reprovide: true,

1040 + name: "LegacyProvider",

1041 + sweep: false,

1042 apply: func(n *harness.Node) {

1043 n.SetIPFSConfig("Provide.DHT.SweepEnabled", false)

1044 },

1045 + // `routing reprovide` blocks until the cycle finishes.

1046 + // minCIDs is ignored (legacy has no stat counter).

1047 + awaitReprovide: func(t *testing.T, n *harness.Node, minCIDs int64) int64 {

1048 + n.IPFS("routing", "reprovide")

1049 + return minCIDs

1050 + },

1051 },

1052 {

749 - name: "SweepingProvider",

750 - reprovide: false,

1053 + name: "SweepingProvider",

1054 + sweep: true,

1055 apply: func(n *harness.Node) {

1056 n.SetIPFSConfig("Provide.DHT.SweepEnabled", true)

1057 },

1058 + // No manual trigger exists for sweep. Poll `provide stat`

1059 + // until the reprovide cycle completes.

1060 + awaitReprovide: func(t *testing.T, n *harness.Node, minCIDs int64) int64 {

1061 + // 90s accounts for provider bootstrap time (connecting

1062 + // to ephemeral peers, measuring prefix length) before

1063 + // the 30s reprovide cycle starts. On CI with parallel

1064 + // tests, bootstrap can take 20-30s.

1065 + return waitForSweepReprovide(t, n, 90*time.Second, minCIDs)

1066 + },

1067 },

1068 }

1069

1070 for _, v := range variants {

1071 t.Run(v.name, func(t *testing.T) {

1072 // t.Parallel()

760 - runProviderSuite(t, v.reprovide, v.apply)

1073 + runProviderSuite(t, v.sweep, v.apply, v.awaitReprovide)

1074

1075 // Resume tests only apply to SweepingProvider

763 - if v.name == "SweepingProvider" {

1076 + if v.sweep {

1077 runResumeTests(t, v.apply)

1078 }

1079 })

1080 }

1081 }

1082

1083 +// TestProviderUniqueDedupLogging verifies that the +unique bloom filter

1084 +// deduplication produces a "skippedBranches" log with a value > 0 when

1085 +// two pins share content. Tests both the fast-provide-dag path (immediate

1086 +// provide on pin add) and the reprovide cycle path.

1087 +func TestProviderUniqueDedupLogging(t *testing.T) {

1088 + t.Parallel()

1089 +

1090 + // Shared data that both pins will reference. Two pins containing

1091 + // the same file block give the bloom something to dedup.

1092 + sharedData := random.Bytes(10 * 1024) // 10 KiB, single block

1093 +

1094 + t.Run("fast-provide-dag dedup across pins in single call", func(t *testing.T) {

1095 + t.Parallel()

1096 +

1097 + h := harness.NewT(t)

1098 + node := h.NewNode().Init()

1099 + node.SetIPFSConfig("Provide.Strategy", "pinned+unique")

1100 + node.SetIPFSConfig("Provide.DHT.SweepEnabled", true)

1101 + node.SetIPFSConfig("Import.UnixFSChunker", "size-5120") // 5 KiB chunks

1102 + h.BootstrapWithStubDHT(harness.Nodes{node})

1103 +

1104 + node.StartDaemonWithReq(harness.RunRequest{

1105 + CmdOpts: []harness.CmdOpt{

1106 + harness.RunWithEnv(map[string]string{

1107 + // dagwalker: bloom creation log

1108 + // core/commands/cmdenv: fast-provide-dag finished log

1109 + "GOLOG_LOG_LEVEL": "error,dagwalker=info,core/commands/cmdenv=info",

1110 + }),

1111 + },

1112 + }, "")

1113 + defer node.StopDaemon()

1114 +

1115 + // 10 KiB file with 5 KiB chunks = 1 file root + 2 chunks = 3 blocks.

1116 + // Two dirs each containing the file under different names:

1117 + // dirA/fileA → same 3 blocks

1118 + // dirB/fileB → same 3 blocks

1119 + // Pinning both in a single `pin add` shares one bloom tracker.

1120 + // Walking dirA: dirA + file root + chunk1 + chunk2 = 4 provided.

1121 + // Walking dirB: dirB + file root (bloom hit, skip subtree) = 1 provided, 1 skipped.

1122 + // Total: 5 provided, 1 skipped branch (file root in dirB; its

1123 + // 2 chunks are never visited because the parent was skipped).

1124 + cidFile := node.IPFSAdd(bytes.NewReader(sharedData), "-Q", "--pin=false")

1125 + node.IPFS("files", "mkdir", "-p", "/dirA")

1126 + node.IPFS("files", "cp", "/ipfs/"+cidFile, "/dirA/fileA")

1127 + cidDirA := node.IPFS("files", "stat", "--hash", "/dirA").Stdout.Trimmed()

1128 + node.IPFS("files", "mkdir", "-p", "/dirB")

1129 + node.IPFS("files", "cp", "/ipfs/"+cidFile, "/dirB/fileB")

1130 + cidDirB := node.IPFS("files", "stat", "--hash", "/dirB").Stdout.Trimmed()

1131 + require.NotEqual(t, cidDirA, cidDirB, "dirs must differ to test dedup")

1132 + // Single pin add with both CIDs shares one bloom.

1133 + node.IPFS("pin", "add", "--fast-provide-dag", "--fast-provide-wait", cidDirA, cidDirB)

1134 +

1135 + daemonLog := node.Daemon.Stderr.String()

1136 + require.Contains(t, daemonLog, "bloom tracker created")

1137 + require.NotContains(t, daemonLog, "bloom tracker autoscaled")

1138 + require.Contains(t, daemonLog, `"providedCIDs": 5`)

1139 + require.Contains(t, daemonLog, `"skippedBranches": 1`)

1140 + })

1141 +

1142 + t.Run("reprovide cycle dedup across pins", func(t *testing.T) {

1143 + t.Parallel()

1144 +

1145 + h := harness.NewT(t)

1146 + nodes := h.NewNodes(2).Init()

1147 + for _, n := range nodes {

1148 + n.SetIPFSConfig("Provide.Strategy", "pinned+unique")

1149 + n.SetIPFSConfig("Provide.DHT.SweepEnabled", true)

1150 + n.SetIPFSConfig("Import.UnixFSChunker", "size-5120") // 5 KiB chunks

1151 + }

1152 + publisher := nodes[0]

1153 + publisher.SetIPFSConfig("Provide.DHT.Interval", "30s")

1154 + h.BootstrapWithStubDHT(nodes)

1155 +

1156 + // Same file structure as fast-provide-dag test above.

1157 + // The reprovide cycle walks all recursive pins:

1158 + // pin dirA: dirA + file root + chunk1 + chunk2 = 4 provided

1159 + // pin empty MFS root (always present): 1 provided

1160 + // pin dirB: dirB + file root (bloom hit, skip subtree) = 1 provided, 1 skipped

1161 + // Total: 6 provided, 1 skipped branch.

1162 + cidFile := publisher.IPFSAdd(bytes.NewReader(sharedData), "-Q", "--pin=false")

1163 + publisher.IPFS("files", "mkdir", "-p", "/dirA")

1164 + publisher.IPFS("files", "cp", "/ipfs/"+cidFile, "/dirA/fileA")

1165 + cidDirA := publisher.IPFS("files", "stat", "--hash", "/dirA").Stdout.Trimmed()

1166 + publisher.IPFS("pin", "add", cidDirA)

1167 + publisher.IPFS("files", "mkdir", "-p", "/dirB")

1168 + publisher.IPFS("files", "cp", "/ipfs/"+cidFile, "/dirB/fileB")

1169 + cidDirB := publisher.IPFS("files", "stat", "--hash", "/dirB").Stdout.Trimmed()

1170 + require.NotEqual(t, cidDirA, cidDirB, "dirs must differ to test dedup")

1171 + publisher.IPFS("pin", "add", cidDirB)

1172 +

1173 + nodes[0].StartDaemonWithReq(harness.RunRequest{

1174 + CmdOpts: []harness.CmdOpt{

1175 + harness.RunWithEnv(map[string]string{

1176 + "GOLOG_LOG_LEVEL": "error,dagwalker=info,core:constructor=info",

1177 + }),

1178 + },

1179 + }, "")

1180 + nodes[1].StartDaemon()

1181 + defer nodes.StopDaemons()

1182 + nodes.Connect()

1183 +

1184 + waitForSweepReprovide(t, publisher, 90*time.Second, 6)

1185 +

1186 + daemonLog := publisher.Daemon.Stderr.String()

1187 + require.Contains(t, daemonLog, "bloom tracker created")

1188 + require.NotContains(t, daemonLog, "bloom tracker autoscaled")

1189 + require.Contains(t, daemonLog, `"providedCIDs": 6`)

1190 + require.Contains(t, daemonLog, `"skippedBranches": 1`)

1191 + })

1192 +}

1193 +

1194 +// TestProviderFastProvideDAGAsyncSurvives verifies that

1195 +// --fast-provide-dag without --fast-provide-wait runs a background

1196 +// DAG walk that outlives the command handler and publishes every

1197 +// block of the newly added DAG to the routing system.

1198 +//

1199 +// The async walk runs in a goroutine parented on the IpfsNode

1200 +// lifetime context (not req.Context), so it keeps running after

1201 +// `ipfs add` returns and is only cancelled on daemon shutdown.

1202 +//

1203 +// Provide.DHT.Interval is set high so the scheduled reprovide

1204 +// cycle cannot fire during the test window. That makes the async

1205 +// walk the only path that can publish non-root block CIDs.

1206 +func TestProviderFastProvideDAGAsyncSurvives(t *testing.T) {

1207 + t.Parallel()

1208 +

1209 + h := harness.NewT(t)

1210 + nodes := h.NewNodes(2).Init()

1211 + for _, n := range nodes {

1212 + n.SetIPFSConfig("Provide.Strategy", "pinned")

1213 + n.SetIPFSConfig("Provide.DHT.SweepEnabled", true)

1214 + // Small chunks so a modest file produces many leaf blocks.

1215 + n.SetIPFSConfig("Import.UnixFSChunker", "size-1024")

1216 + }

1217 + publisher, peers := nodes[0], nodes[1:]

1218 + publisher.SetIPFSConfig("Provide.DHT.Interval", "1h")

1219 + h.BootstrapWithStubDHT(nodes)

1220 +

1221 + publisher.StartDaemonWithReq(harness.RunRequest{

1222 + CmdOpts: []harness.CmdOpt{

1223 + harness.RunWithEnv(map[string]string{

1224 + "GOLOG_LOG_LEVEL": "error,core/commands/cmdenv=info",

1225 + }),

1226 + },

1227 + }, "")

1228 + nodes[1].StartDaemon()

1229 + defer nodes.StopDaemons()

1230 + nodes.Connect()

1231 +

1232 + // 16 KiB + 1 KiB chunks yields a file root plus many leaf

1233 + // blocks, so the providedCIDs count after the walk is

1234 + // unambiguous.

1235 + data := random.Bytes(16 * 1024)

1236 + cidFile := publisher.IPFSAdd(bytes.NewReader(data), "-Q",

1237 + "--pin=true",

1238 + "--fast-provide-dag=true",

1239 + // --fast-provide-wait deliberately omitted: the walk

1240 + // runs in the background after `ipfs add` returns.

1241 + )

1242 +

1243 + // Pull a chunk CID out of the file DAG. Chunks are not pin

1244 + // roots, so fast-provide-root does not touch them; only the

1245 + // DAG walk can announce them.

1246 + dagOut := publisher.IPFS("dag", "get", cidFile)

1247 + var dagNode struct {

1248 + Links []struct {

1249 + Hash map[string]string `json:"Hash"`

1250 + } `json:"Links"`

1251 + }

1252 + require.NoError(t, json.Unmarshal(dagOut.Stdout.Bytes(), &dagNode))

1253 + require.Greater(t, len(dagNode.Links), 1, "file should have multiple chunks")

1254 + cidChunk := dagNode.Links[0].Hash["/"]

1255 + require.NotEmpty(t, cidChunk)

1256 +

1257 + // The async walk logs "fast-provide-dag: finished" with a

1258 + // providedCIDs count on completion. A full walk of this file

1259 + // visits the root plus every leaf chunk, so the count is much

1260 + // larger than 2.

1261 + providedRe := regexp.MustCompile(`"providedCIDs": (\d+)`)

1262 + var providedCount int

1263 + require.Eventually(t, func() bool {

1264 + m := providedRe.FindStringSubmatch(publisher.Daemon.Stderr.String())

1265 + if len(m) != 2 {

1266 + return false

1267 + }

1268 + n, err := strconv.Atoi(m[1])

1269 + if err != nil {

1270 + return false

1271 + }

1272 + providedCount = n

1273 + return true

1274 + }, 30*time.Second, 200*time.Millisecond, "async fast-provide-dag walk did not log 'finished'")

1275 +

1276 + require.Greater(t, providedCount, 2,

1277 + "providedCIDs=%d is too small for a full walk of the file DAG", providedCount)

1278 +

1279 + // End-to-end: the peer can find the publisher as a provider

1280 + // for a chunk CID, which only the async walk could have

1281 + // announced within the test window.

1282 + pid := publisher.PeerID().String()

1283 + var found bool

1284 + for _, peer := range peers {

1285 + for i := time.Duration(0); i*timeStep < timeout; i++ {

1286 + res := peer.IPFS("routing", "findprovs", "-n=1", cidChunk)

1287 + if res.Stdout.Trimmed() == pid {

1288 + found = true

1289 + break

1290 + }

1291 + }

1292 + }

1293 + require.True(t, found, "chunk %s not announced by the async walk", cidChunk)

1294 +}

1295 +

1296 // TestHTTPOnlyProviderWithSweepEnabled tests that provider records are correctly

1297 // sent to HTTP routers when Routing.Type="custom" with only HTTP routers configured,

1298 // even when Provide.DHT.SweepEnabled=true (the default since v0.39).

test/dependencies/go.mod

+1 -1

@@ -183,7 +183,7 @@ require (

183 github.com/libp2p/go-flow-metrics v0.3.0 // indirect

184 github.com/libp2p/go-libp2p v0.48.0 // indirect

185 github.com/libp2p/go-libp2p-asn-util v0.4.1 // indirect

186 - github.com/libp2p/go-libp2p-kad-dht v0.39.0 // indirect

186 + github.com/libp2p/go-libp2p-kad-dht v0.39.1-0.20260326020727-bcbc21e9f633 // indirect

187 github.com/libp2p/go-libp2p-kbucket v0.8.0 // indirect

188 github.com/libp2p/go-libp2p-record v0.3.1 // indirect

189 github.com/libp2p/go-libp2p-routing-helpers v0.7.5 // indirect

test/dependencies/go.sum

+2 -2

@@ -582,8 +582,8 @@ github.com/libp2p/go-libp2p v0.48.0 h1:h2BrLAgrj7X8bEN05K7qmrjpNHYA+6tnsGRdprjTn

582 github.com/libp2p/go-libp2p v0.48.0/go.mod h1:Q1fBZNdmC2Hf82husCTfkKJVfHm2we5zk+NWmOGEmWk=

583 github.com/libp2p/go-libp2p-asn-util v0.4.1 h1:xqL7++IKD9TBFMgnLPZR6/6iYhawHKHl950SO9L6n94=

584 github.com/libp2p/go-libp2p-asn-util v0.4.1/go.mod h1:d/NI6XZ9qxw67b4e+NgpQexCIiFYJjErASrYW4PFDN8=

585 -github.com/libp2p/go-libp2p-kad-dht v0.39.0 h1:mww38eBYiUvdsu+Xl/GLlBC0Aa8M+5HAwvafkFOygAM=

586 -github.com/libp2p/go-libp2p-kad-dht v0.39.0/go.mod h1:Po2JugFEkDq9Vig/JXtc153ntOi0q58o4j7IuITCOVs=

585 +github.com/libp2p/go-libp2p-kad-dht v0.39.1-0.20260326020727-bcbc21e9f633 h1:PcubpdBr1BBg39st+CqGp3EOX++DOBK6B/s07P31eMg=

586 +github.com/libp2p/go-libp2p-kad-dht v0.39.1-0.20260326020727-bcbc21e9f633/go.mod h1:Po2JugFEkDq9Vig/JXtc153ntOi0q58o4j7IuITCOVs=

587 github.com/libp2p/go-libp2p-kbucket v0.8.0 h1:QAK7RzKJpYe+EuSEATAaaHYMYLkPDGC18m9jxPLnU8s=

588 github.com/libp2p/go-libp2p-kbucket v0.8.0/go.mod h1:JMlxqcEyKwO6ox716eyC0hmiduSWZZl6JY93mGaaqc4=

589 github.com/libp2p/go-libp2p-record v0.3.1 h1:cly48Xi5GjNw5Wq+7gmjfBiG9HCzQVkiZOUZ8kUl+Fg=