feat(dag): add --local-only to dag export and import (#11229)
* feat(dag): add --local-only to dag export and import - Export: only export blocks present locally; skip missing (partial CAR). --local-only with --offline. Support both binary and base58 link keys. - Import: support partial CARs; --local-only with -- pin-roots=false (error if both --pin-roots and --local-only set). - Fix cidFromBinString to accept base58 key format from link implementations. Signed-off-by: Chayan Das <01chayandas@gmail.com> * chore(deps): update go-car/v2 to latest master - remove local replace directive for go-car/v2 - upgrade to v2.16.1-0.20260306172652-7d2f4aceb070 * fix(dag): avoid CID round-trip in export and fix ci failure Signed-off-by: Chayan Das <01chayandas@gmail.com> * dag: add validation and tests for --local-only flag Signed-off-by: Chayan Das <01chayandas@gmail.com> * chore(deps): bump go-car/v2 to latest master * feat(dag): --local-only auto-sets companion flags Pass --local-only without pairing it with --offline (export) or --pin-roots=false (import); the companion is now implicit. Explicit opposites (--offline=false, --pin-roots=true) are rejected so the intent stays unambiguous. * export: imply --offline so missing blocks are not fetched over the network, which would defeat --local-only * import: imply --pin-roots=false since a partial CAR has no full DAG to pin * tests: cover the new implications and the rejected explicit-opposite combinations; drop the brittle exec.CommandContext path in favor of the existing harness * refactor(dag): use boxo/walker for --local-only export The --local-only branch now uses walker.WalkDAG with WithLocality(bs.Has) and carstorage.NewWritable, matching the MFS+unique provider in core/node/provider.go. Semantics: any input-side read error during the walk (missing block, decode failure, post-locality race) is treated as "not available locally" and the block plus its subtree are skipped. Output-side errors (writable.Put) are still surfaced. --help is updated to call out the best-effort nature. The non-local-only path is unchanged. * test(dag): tighten --local-only tests, add subtree-skip case Pin chunker and max-file-links via a shared shallowDAGArgs so block counts are deterministic regardless of Import.* defaults or active profiles. Tighten existing assertions: * TestDagExportLocalOnly: assert exact fullCount=3 and partialCount=fullCount-1 instead of partialCount<fullCount * TestDagExportLocalOnlyImpliesOffline: assert exact partial block count, not just file Size > 0 (proves --offline was applied) Add TestDagExportLocalOnlySkipsSubtree: builds a 259-block DAG with depth>1 (256 chunks under 2 intermediates), removes an intermediate, and verifies the partial CAR is missing the intermediate plus all 174 of its descendants. Existing tests only exercised leaf removal. Extract countCARBlocks and makePartialDAG helpers used across tests. * docs: changelog entry for --local-only dag export/import * refactor(dag): wrap API explicitly for --local-only Replace the req.Options["offline"] = true mutation with an explicit api.WithOptions(options.Api.Offline(true)) wrap after GetApi, matching the pattern already used in core/commands/dag/import.go. Clarify in comments that the walker reads from the raw blockstore (not via the kubo CoreAPI or DAGService) and therefore cannot trigger a network fetch by construction. The --offline implication exists for api.Block().Stat path resolution, not for the DAG walk itself. * fix(provider): quiet context.Canceled on shutdown ResetCids returns ctx.Err() straight from its ctx-done select, so a shutdown-during-sync surfaces as err="context canceled" while the outer ctx.Err() check at the classifier sometimes races behind the propagation and logs at Error. Classify context.Canceled the same way as keystore.ErrClosed so the message lands at Debug. Applied to both the startup and periodic classifiers. DeadlineExceeded is intentionally not included: nothing in the current call chain imposes a deadline, and a future timeout would be a real failure worth logging at Error. Closes the flake in TestProviderKeystoreSyncShutdownQuiet (10/10 local soak now green; CI hit the race 3 reruns in a row). --------- Signed-off-by: Chayan Das <01chayandas@gmail.com> Co-authored-by: Marcin Rataj <lidel@lidel.org>
Chayan Das committed
May 26, 2026 at 03:54 UTC
0acc68868ea2690fdb42ffd406fcac6af29c1d9a
13 files changed
+424
-21
core/commands/dag/dag.go
+12
-1
@@ -24,6 +24,7 @@ const (
24
fastProvideRootOptionName = "fast-provide-root"
25
fastProvideDAGOptionName = "fast-provide-dag"
26
fastProvideWaitOptionName = "fast-provide-wait"
27
+ localOnlyOptionName = "local-only"
28
)
29
30
// DagCmd provides a subset of commands for interacting with ipld dag objects
@@ -193,6 +194,10 @@ Note:
194
currently present in the blockstore does not represent a complete DAG,
195
pinning of that individual root will fail.
196
197
+ Use --local-only to import a partial CAR (e.g. from 'dag export
198
+ --local-only'). --local-only implies --pin-roots=false because a partial
199
+ CAR has no full DAG to pin.
200
+
201
FAST PROVIDE OPTIMIZATION:
202
203
Root CIDs from CAR headers are immediately provided to the DHT in addition
@@ -213,7 +218,8 @@ Specification of CAR formats: https://ipld.io/specs/transport/car/
218
cmds.FileArg("path", true, true, "The path of a .car file.").EnableStdin(),
219
},
220
Options: []cmds.Option{
216
- cmds.BoolOption(pinRootsOptionName, "Pin optional roots listed in the .car headers after importing.").WithDefault(true),
221
+ cmds.BoolOption(pinRootsOptionName, "Pin optional roots listed in the .car headers after importing. Default: true."),
222
+ cmds.BoolOption(localOnlyOptionName, "Import a partial CAR (e.g. from 'dag export --local-only'). Implies --pin-roots=false."),
223
cmds.BoolOption(silentOptionName, "No output."),
224
cmds.BoolOption(statsOptionName, "Output stats."),
225
cmds.BoolOption(fastProvideRootOptionName, "Immediately provide root CIDs to DHT in addition to regular queue, for faster discovery. Default: Import.FastProvideRoot"),
@@ -277,6 +283,10 @@ var DagExportCmd = &cmds.Command{
283
Note that at present only single root selections / .car files are supported.
284
The output of blocks happens in strict DAG-traversal, first-seen, order.
285
CAR file follows the CARv1 format: https://ipld.io/specs/transport/car/carv1/
286
+
287
+Use --local-only for a best-effort export from the local blockstore: blocks
288
+that are missing or unreadable locally (and their subtrees) are skipped, so
289
+the resulting CAR is partial. --local-only implies --offline.
290
`,
291
HTTP: &cmds.HTTPHelpText{
292
ResponseContentType: "application/vnd.ipld.car",
@@ -287,6 +297,7 @@ CAR file follows the CARv1 format: https://ipld.io/specs/transport/car/carv1/
297
},
298
Options: []cmds.Option{
299
cmds.BoolOption(progressOptionName, "p", "Stream progress data. Defaults to true when stderr is a terminal."),
300
+ cmds.BoolOption(localOnlyOptionName, "Best-effort export of locally-available blocks; missing or unreadable blocks (and their subtrees) are skipped. Implies --offline."),
301
},
302
Run: dagExport,
303
PostRun: cmds.PostRunMap{
core/commands/dag/export.go
+89
-1
@@ -9,13 +9,17 @@ import (
9
"time"
10
11
"github.com/cheggaaa/pb/v3"
12
+ blockstore "github.com/ipfs/boxo/blockstore"
13
+ "github.com/ipfs/boxo/dag/walker"
14
cid "github.com/ipfs/go-cid"
15
cmds "github.com/ipfs/go-ipfs-cmds"
16
ipld "github.com/ipfs/go-ipld-format"
17
"github.com/ipfs/kubo/core/commands/cmdenv"
18
"github.com/ipfs/kubo/core/commands/cmdutils"
19
iface "github.com/ipfs/kubo/core/coreiface"
20
+ "github.com/ipfs/kubo/core/coreiface/options"
21
gocar "github.com/ipld/go-car/v2"
22
+ carstorage "github.com/ipld/go-car/v2/storage"
23
cidlink "github.com/ipld/go-ipld-prime/linking/cid"
24
selectorparse "github.com/ipld/go-ipld-prime/traversal/selector/parse"
25
)
@@ -34,10 +38,28 @@ func dagExport(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment
38
return err
39
}
40
41
+ localOnly, _ := req.Options[localOnlyOptionName].(bool)
42
+ if localOnly {
43
+ // --local-only and --offline=false contradict each other.
44
+ if offline, set := req.Options["offline"].(bool); set && !offline {
45
+ return fmt.Errorf("--%s implies --offline and cannot be combined with --offline=false; please drop one of them", localOnlyOptionName)
46
+ }
47
+ }
48
+
49
api, err := cmdenv.GetApi(env, req)
50
if err != nil {
51
return err
52
}
53
+ if localOnly {
54
+ // --local-only implies --offline so api.Block().Stat below cannot
55
+ // reach out for path resolution. The DAG walk itself uses the raw
56
+ // blockstore via walker (see exportPartialCAR) and is local by
57
+ // construction regardless of this setting.
58
+ api, err = api.WithOptions(options.Api.Offline(true))
59
+ if err != nil {
60
+ return err
61
+ }
62
+ }
63
64
// Resolve path and confirm the root block is available, fail fast if not
65
b, err := api.Block().Stat(req.Context, p)
@@ -46,6 +68,15 @@ func dagExport(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment
68
}
69
c := b.Path().RootCid()
70
71
+ var bs blockstore.Blockstore
72
+ if localOnly {
73
+ node, err := cmdenv.GetNode(env)
74
+ if err != nil {
75
+ return err
76
+ }
77
+ bs = node.Blockstore
78
+ }
79
+
80
pipeR, pipeW := io.Pipe()
81
82
errCh := make(chan error, 2) // we only report the 1st error
@@ -57,6 +88,13 @@ func dagExport(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment
88
close(errCh)
89
}()
90
91
+ if localOnly {
92
+ if err := exportPartialCAR(req.Context, bs, c, pipeW); err != nil {
93
+ errCh <- err
94
+ }
95
+ return
96
+ }
97
+
98
lsys := cidlink.DefaultLinkSystem()
99
lsys.SetReadStorage(&dagStore{dag: api.Dag(), ctx: req.Context})
100
@@ -105,6 +143,56 @@ func dagExport(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment
143
return err
144
}
145
146
+// exportPartialCAR is the best-effort engine behind `dag export --local-only`.
147
+// It walks the DAG rooted at root and writes the visited blocks to w as a
148
+// CARv1 stream.
149
+//
150
+// The walker reads from the raw blockstore directly (not via the kubo
151
+// CoreAPI or DAGService), so it is structurally incapable of triggering a
152
+// network fetch. Any block missing or unreadable locally, plus its entire
153
+// subtree, is silently skipped: the resulting CAR is partial by design.
154
+//
155
+// Errors writing the CAR itself (emit failures) are surfaced: those are
156
+// output problems, not local-availability problems.
157
+//
158
+// This mirrors the MFS+unique provider in core/node/provider.go.
159
+func exportPartialCAR(ctx context.Context, bs blockstore.Blockstore, root cid.Cid, w io.Writer) error {
160
+ writable, err := carstorage.NewWritable(w, []cid.Cid{root}, gocar.WriteAsCarV1(true))
161
+ if err != nil {
162
+ return err
163
+ }
164
+
165
+ // Capture the first emit (write-side) error so the walk stops cleanly.
166
+ var emitErr error
167
+ emit := func(k cid.Cid) bool {
168
+ blk, err := bs.Get(ctx, k)
169
+ if err != nil {
170
+ // Any read error after locality passed (e.g. GC race or
171
+ // corruption) is treated as "not available locally": skip
172
+ // the block and keep streaming the rest of the partial CAR.
173
+ return true
174
+ }
175
+ if err := writable.Put(ctx, k.KeyString(), blk.RawData()); err != nil {
176
+ emitErr = err
177
+ return false
178
+ }
179
+ return true
180
+ }
181
+
182
+ // Both the locality check (bs.Has) and the link fetcher read straight
183
+ // from the blockstore, so the walk cannot reach the network. Errors
184
+ // inside walker (locality, fetch) are skip-and-log, matching the
185
+ // best-effort semantics here.
186
+ if err := walker.WalkDAG(ctx, root,
187
+ walker.LinksFetcherFromBlockstore(bs),
188
+ emit,
189
+ walker.WithLocality(func(ctx context.Context, k cid.Cid) (bool, error) { return bs.Has(ctx, k) }),
190
+ ); err != nil {
191
+ return err
192
+ }
193
+ return emitErr
194
+}
195
+
196
func finishCLIExport(res cmds.Response, re cmds.ResponseEmitter) error {
197
if !cmdenv.ShouldShowProgress(res.Request(), progressOptionName) {
198
return cmds.Copy(re, res)
@@ -185,7 +273,7 @@ func cidFromBinString(key string) (cid.Cid, error) {
273
return cid.Undef, fmt.Errorf("dagStore: key was not a cid: %w", err)
274
}
275
if l != len(key) {
188
- return cid.Undef, fmt.Errorf("dagSore: key was not a cid: had %d bytes leftover", len(key)-l)
276
+ return cid.Undef, fmt.Errorf("dagStore: key was not a cid: had %d bytes leftover", len(key)-l)
277
}
278
return k, nil
279
}
core/commands/dag/import.go
+17
-1
@@ -48,8 +48,24 @@ func dagImport(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment
48
return err
49
}
50
51
- doPinRoots, _ := req.Options[pinRootsOptionName].(bool)
51
+ pinRootsVal, pinRootsSet := req.Options[pinRootsOptionName].(bool)
52
+ localOnly, _ := req.Options[localOnlyOptionName].(bool)
53
+
54
+ // --pin-roots defaults to true; the default is applied here (not via
55
+ // .WithDefault) so we can tell apart "user explicitly passed true" from
56
+ // "no value provided".
57
+ doPinRoots := true
58
+ if pinRootsSet {
59
+ doPinRoots = pinRootsVal
60
+ }
61
62
+ if localOnly {
63
+ if pinRootsSet && pinRootsVal {
64
+ return fmt.Errorf("--%s implies --%s=false and cannot be combined with --%s=true; please drop one of them", localOnlyOptionName, pinRootsOptionName, pinRootsOptionName)
65
+ }
66
+ // --local-only implies --pin-roots=false: a partial CAR has no full DAG to pin.
67
+ doPinRoots = false
68
+ }
69
fastProvideRoot, fastProvideRootSet := req.Options[fastProvideRootOptionName].(bool)
70
fastProvideDAG, fastProvideDAGSet := req.Options[fastProvideDAGOptionName].(bool)
71
fastProvideWait, fastProvideWaitSet := req.Options[fastProvideWaitOptionName].(bool)
core/node/provider.go
+12
-6
@@ -880,11 +880,13 @@ func SweepingProviderOpt(cfg *config.Config) fx.Option {
880
strategy := cfg.Provide.Strategy.WithDefault(config.DefaultProvideStrategy)
881
providerLog.Infow("provider keystore sync started", "strategy", strategy)
882
if err := syncKeystore(ctx); err != nil {
883
- // ErrClosed means the keystore was closed by the shutdown
884
- // hook while this goroutine was still in flight: the
885
- // OnStart ctx is not cancelled yet, so we classify the
886
- // failure as shutdown explicitly.
887
- if ctx.Err() != nil || errors.Is(err, keystore.ErrClosed) {
883
+ // Shutdown can race ahead of ctx.Err() becoming
884
+ // visible here: ResetCids returns ctx.Err()
885
+ // straight from its own ctx-done select, and the
886
+ // keystore can also close mid-sync (ErrClosed)
887
+ // before the OnStart ctx is cancelled. Classify
888
+ // both as shutdown.
889
+ if ctx.Err() != nil || errors.Is(err, context.Canceled) || errors.Is(err, keystore.ErrClosed) {
890
providerLog.Debugw("provider keystore sync interrupted by shutdown", "err", err, "strategy", strategy)
891
} else {
892
providerLog.Errorw("provider keystore sync failed", "err", err, "strategy", strategy)
@@ -908,7 +910,11 @@ func SweepingProviderOpt(cfg *config.Config) fx.Option {
910
return
911
case <-ticker.C:
912
if err := syncKeystore(gcCtx); err != nil {
911
- if gcCtx.Err() != nil || errors.Is(err, keystore.ErrClosed) {
913
+ // See classifier note on the startup-sync
914
+ // branch above: context.Canceled can
915
+ // arrive ahead of gcCtx.Err() becoming
916
+ // visible to this goroutine.
917
+ if gcCtx.Err() != nil || errors.Is(err, context.Canceled) || errors.Is(err, keystore.ErrClosed) {
918
providerLog.Debugw("provider keystore sync interrupted by shutdown", "err", err)
919
} else {
920
providerLog.Errorw("provider keystore sync failed", "err", err)
docs/changelogs/v0.42.md
+12
@@ -11,6 +11,7 @@ This release was brought to you by the [Shipyard](https://ipshipyard.com/) team.
11
- [Overview](#overview)
12
- [🔦 Highlights](#-highlights)
13
- [🎯 Announce CIDs on demand with `ipfs provide once`](#-announce-cids-on-demand-with-ipfs-provide-once)
14
+ - [🧩 Export and import partial CARs with `--local-only`](#-export-and-import-partial-cars-with---local-only)
15
- [⚙️ `Provide.DHT.Interval=0` no longer disables providing](#%EF%B8%8F-providedhtinterval0-no-longer-disables-providing)
16
- [🐛 Fixed pin operations hanging under pinned reprovide strategies](#-fixed-pin-operations-hanging-under-pinned-reprovide-strategies)
17
- [🐛 Smoother first-run upgrades from very old repos](#-smoother-first-run-upgrades-from-very-old-repos)
@@ -46,6 +47,17 @@ In a terminal, the command shows a running count of queued CIDs. With `--enc=jso
47
48
`ipfs routing provide` keeps working but is deprecated. See `ipfs provide once --help` for usage and migration notes.
49
50
+#### 🧩 Export and import partial CARs with `--local-only`
51
+
52
+`ipfs dag export --local-only` writes a CAR with only the blocks you have locally; any missing blocks (and their subtrees) are skipped instead of failing the export. `ipfs dag import --local-only` reads such a partial CAR without trying to pin its roots.
53
+
54
+This is useful when:
55
+
56
+- you want to share part of a DAG (for example an MFS tree) that is only partly cached locally
57
+- you fetched a partial CAR from a gateway that supports [IPIP-0402](https://specs.ipfs.tech/ipips/ipip-0402/) and want to add what you got to your local store
58
+
59
+`--local-only` sets the matching companion flag automatically: on export it implies `--offline`; on import it implies `--pin-roots=false`. See `ipfs dag export --help` and `ipfs dag import --help` for details.
60
+
61
#### ⚙️ `Provide.DHT.Interval=0` no longer disables providing
62
63
`Provide.DHT.Interval=0` now disables only the periodic reprovide schedule. New CIDs still announce via fast-provide-root and `ipfs provide once`. To fully disable providing, set [`Provide.Enabled=false`](https://github.com/ipfs/kubo/blob/master/docs/config.md#provideenabled).
docs/examples/kubo-as-a-library/go.mod
+1
-1
@@ -101,7 +101,7 @@ require (
101
github.com/ipfs/go-peertaskqueue v0.8.3 // indirect
102
github.com/ipfs/go-test v0.3.0 // indirect
103
github.com/ipfs/go-unixfsnode v1.10.4 // indirect
104
- github.com/ipld/go-car/v2 v2.16.0 // indirect
104
+ github.com/ipld/go-car/v2 v2.16.1-0.20260428045700-c4b9f366f20c // indirect
105
github.com/ipld/go-codec-dagpb v1.7.0 // indirect
106
github.com/ipld/go-ipld-prime v0.23.0 // indirect
107
github.com/ipshipyard/p2p-forge v0.8.0 // indirect
docs/examples/kubo-as-a-library/go.sum
+2
-2
@@ -422,8 +422,8 @@ github.com/ipfs/go-test v0.3.0 h1:0Y4Uve3tp9HI+2lIJjfOliOrOgv/YpXg/l1y3P4DEYE=
422
github.com/ipfs/go-test v0.3.0/go.mod h1:JK+U8pRpATZb7lsYNSJlCj3WYB3cFfWIbI6nWRM/GFk=
423
github.com/ipfs/go-unixfsnode v1.10.4 h1:cMmMyOrSjQkPVQbQvt8trErIn6jhayNf9pBA9oOwfxY=
424
github.com/ipfs/go-unixfsnode v1.10.4/go.mod h1:Vu1e/s7ToALBBRo38sJ8DwUVWmSeQMTdxk5/rcHl7d0=
425
-github.com/ipld/go-car/v2 v2.16.0 h1:LWe0vmN/QcQmUU4tr34W5Nv5mNraW+G6jfN2s+ndBco=
426
-github.com/ipld/go-car/v2 v2.16.0/go.mod h1:RqFGWN9ifcXVmCrTAVnfnxiWZk1+jIx67SYhenlmL34=
425
+github.com/ipld/go-car/v2 v2.16.1-0.20260428045700-c4b9f366f20c h1:ZFONxHSj6bzzB9eKIu+yS2AazTJe7j9FPesfy4sZSE0=
426
+github.com/ipld/go-car/v2 v2.16.1-0.20260428045700-c4b9f366f20c/go.mod h1:/4HY8tFZ1q42Mw54ILLPQfjkUqMJxFKqY1yMDKHlYko=
427
github.com/ipld/go-codec-dagpb v1.7.0 h1:hpuvQjCSVSLnTnHXn+QAMR0mLmb1gA6wl10LExo2Ts0=
428
github.com/ipld/go-codec-dagpb v1.7.0/go.mod h1:rD3Zg+zub9ZnxcLwfol/OTQRVjaLzXypgy4UqHQvilM=
429
github.com/ipld/go-ipld-prime v0.11.0/go.mod h1:+WIAkokurHmZ/KwzDOMUuoeJgaRQktHtEaLglS3ZeV8=
go.mod
+1
-1
@@ -43,7 +43,7 @@ require (
43
github.com/ipfs/go-metrics-prometheus v0.1.0
44
github.com/ipfs/go-test v0.3.0
45
github.com/ipfs/go-unixfsnode v1.10.4
46
- github.com/ipld/go-car/v2 v2.16.0
46
+ github.com/ipld/go-car/v2 v2.16.1-0.20260428045700-c4b9f366f20c
47
github.com/ipld/go-codec-dagpb v1.7.0
48
github.com/ipld/go-ipld-prime v0.23.0
49
github.com/ipshipyard/p2p-forge v0.8.0
go.sum
+2
-2
@@ -465,8 +465,8 @@ github.com/ipfs/go-test v0.3.0 h1:0Y4Uve3tp9HI+2lIJjfOliOrOgv/YpXg/l1y3P4DEYE=
465
github.com/ipfs/go-test v0.3.0/go.mod h1:JK+U8pRpATZb7lsYNSJlCj3WYB3cFfWIbI6nWRM/GFk=
466
github.com/ipfs/go-unixfsnode v1.10.4 h1:cMmMyOrSjQkPVQbQvt8trErIn6jhayNf9pBA9oOwfxY=
467
github.com/ipfs/go-unixfsnode v1.10.4/go.mod h1:Vu1e/s7ToALBBRo38sJ8DwUVWmSeQMTdxk5/rcHl7d0=
468
-github.com/ipld/go-car/v2 v2.16.0 h1:LWe0vmN/QcQmUU4tr34W5Nv5mNraW+G6jfN2s+ndBco=
469
-github.com/ipld/go-car/v2 v2.16.0/go.mod h1:RqFGWN9ifcXVmCrTAVnfnxiWZk1+jIx67SYhenlmL34=
468
+github.com/ipld/go-car/v2 v2.16.1-0.20260428045700-c4b9f366f20c h1:ZFONxHSj6bzzB9eKIu+yS2AazTJe7j9FPesfy4sZSE0=
469
+github.com/ipld/go-car/v2 v2.16.1-0.20260428045700-c4b9f366f20c/go.mod h1:/4HY8tFZ1q42Mw54ILLPQfjkUqMJxFKqY1yMDKHlYko=
470
github.com/ipld/go-codec-dagpb v1.7.0 h1:hpuvQjCSVSLnTnHXn+QAMR0mLmb1gA6wl10LExo2Ts0=
471
github.com/ipld/go-codec-dagpb v1.7.0/go.mod h1:rD3Zg+zub9ZnxcLwfol/OTQRVjaLzXypgy4UqHQvilM=
472
github.com/ipld/go-ipld-prime v0.11.0/go.mod h1:+WIAkokurHmZ/KwzDOMUuoeJgaRQktHtEaLglS3ZeV8=
test/cli/dag_test.go
+268
@@ -2,8 +2,11 @@ package cli
2
3
import (
4
"encoding/json"
5
+ "fmt"
6
"io"
7
"os"
8
+ "path/filepath"
9
+ "strings"
10
"testing"
11
"time"
12
@@ -344,3 +347,268 @@ func TestDagImportFastProvide(t *testing.T) {
347
require.Contains(t, daemonLog, "fast-provide-root: skipped")
348
})
349
}
350
+
351
+// dagRefs returns root plus recursive ref CIDs from "ipfs refs -r --unique root".
352
+func dagRefs(node *harness.Node, root string) []string {
353
+ refsRes := node.IPFS("refs", "-r", "--unique", root)
354
+ refs := []string{root}
355
+ for _, line := range testutils.SplitLines(strings.TrimSpace(refsRes.Stdout.String())) {
356
+ if line != "" {
357
+ refs = append(refs, line)
358
+ }
359
+ }
360
+ return refs
361
+}
362
+
363
+// countCARBlocks imports the CAR at carPath onto a fresh node and returns the
364
+// number of blocks reported by `dag import --stats`. The fresh node guarantees
365
+// the count reflects what is in the CAR, not what was already in the store.
366
+func countCARBlocks(t *testing.T, carPath string) int {
367
+ t.Helper()
368
+ node := harness.NewT(t).NewNode().Init().StartDaemon()
369
+ defer node.StopDaemon()
370
+
371
+ car, err := os.Open(carPath)
372
+ require.NoError(t, err)
373
+ defer car.Close()
374
+
375
+ res := node.Runner.Run(harness.RunRequest{
376
+ Path: node.IPFSBin,
377
+ Args: []string{"dag", "import", "--pin-roots=false", "--stats"},
378
+ CmdOpts: []harness.CmdOpt{harness.RunWithStdin(car)},
379
+ })
380
+ require.Equal(t, 0, res.ExitCode(), "dag import --stats failed: %s", res.Stderr.String())
381
+
382
+ var n int
383
+ for _, line := range testutils.SplitLines(res.Stdout.String()) {
384
+ if _, err := fmt.Sscanf(line, "Imported %d blocks", &n); err == nil {
385
+ break
386
+ }
387
+ }
388
+ require.Greater(t, n, 0, "expected 'Imported N blocks' in stdout: %q", res.Stdout.String())
389
+ return n
390
+}
391
+
392
+// shallowDAGArgs are the `ipfs add` args used by the partial-DAG helpers
393
+// below. Chunker and max-file-links are pinned so the resulting DAG shape
394
+// (root + 2 raw leaves) is independent of changes to Import.* defaults or
395
+// applied profiles.
396
+var shallowDAGArgs = []string{"--raw-leaves", "--chunker=size-262144", "--max-file-links=174"}
397
+
398
+// makePartialDAG adds a 300 KiB file with shallowDAGArgs (yielding root + 2
399
+// raw leaves) and then deletes the first leaf so the node holds a DAG with
400
+// one missing block. Returns the root CID and the CID that was removed.
401
+func makePartialDAG(t *testing.T, node *harness.Node, seed string, addArgs ...string) (root, removed string) {
402
+ t.Helper()
403
+ root = node.IPFSAddDeterministic("300KiB", seed, append(shallowDAGArgs, addArgs...)...)
404
+ refs := dagRefs(node, root)
405
+ require.Equal(t, 3, len(refs), "expected exactly root + 2 raw leaves with pinned chunker/max-links, got %v", refs)
406
+ require.Equal(t, 0, node.RunIPFS("pin", "rm", root).ExitCode())
407
+ require.Equal(t, 0, node.RunIPFS("block", "rm", refs[1]).ExitCode())
408
+ return root, refs[1]
409
+}
410
+
411
+// TestDagExportLocalOnly verifies the core promise of --local-only: a DAG
412
+// with a single missing leaf can still be exported as a partial CAR, and
413
+// the partial CAR contains exactly the full DAG minus the removed block.
414
+func TestDagExportLocalOnly(t *testing.T) {
415
+ t.Parallel()
416
+ node := harness.NewT(t).NewNode().Init().StartDaemon()
417
+ defer node.StopDaemon()
418
+
419
+ // Snapshot the full DAG to a CAR before the block is removed, so we
420
+ // have a baseline block count to compare against.
421
+ root := node.IPFSAddDeterministic("300KiB", "dag-export-local-only", shallowDAGArgs...)
422
+ fullCarPath := filepath.Join(node.Dir, "full.car")
423
+ require.NoError(t, node.IPFSDagExport(root, fullCarPath))
424
+ fullCount := countCARBlocks(t, fullCarPath)
425
+ require.Equal(t, 3, fullCount, "expected root + 2 raw leaves (full=%d)", fullCount)
426
+
427
+ // Drop one leaf so the local DAG is partial.
428
+ refs := dagRefs(node, root)
429
+ require.Equal(t, 0, node.RunIPFS("pin", "rm", root).ExitCode())
430
+ require.Equal(t, 0, node.RunIPFS("block", "rm", refs[1]).ExitCode())
431
+
432
+ // Sanity: plain --offline (without --local-only) must fail loudly
433
+ // when a block is missing. This guards the existing behavior.
434
+ res := node.Runner.Run(harness.RunRequest{
435
+ Path: node.IPFSBin,
436
+ Args: []string{"dag", "export", "--offline", root},
437
+ CmdOpts: []harness.CmdOpt{harness.RunWithStdout(io.Discard)},
438
+ })
439
+ require.NotEqual(t, 0, res.ExitCode(), "dag export --offline must fail when a block is missing")
440
+ require.Contains(t, res.Stderr.String(), "block was not found locally")
441
+
442
+ // --local-only must succeed and produce a CAR with exactly the
443
+ // full DAG minus the one removed leaf.
444
+ partialCarPath := filepath.Join(node.Dir, "partial.car")
445
+ require.NoError(t, node.IPFSDagExport(root, partialCarPath, "--local-only", "--offline"))
446
+ partialCount := countCARBlocks(t, partialCarPath)
447
+
448
+ require.Equal(t, fullCount-1, partialCount,
449
+ "partial CAR should be exactly the full DAG minus the one removed leaf (full=%d, partial=%d)",
450
+ fullCount, partialCount)
451
+}
452
+
453
+// TestDagExportLocalOnlyImpliesOffline verifies that --local-only on its own
454
+// makes a partial-DAG export succeed: it implies --offline so the user does
455
+// not have to pass both flags.
456
+func TestDagExportLocalOnlyImpliesOffline(t *testing.T) {
457
+ t.Parallel()
458
+ node := harness.NewT(t).NewNode().Init().StartDaemon()
459
+ defer node.StopDaemon()
460
+
461
+ root, _ := makePartialDAG(t, node, "dag-export-local-only-implies")
462
+
463
+ // Export with only --local-only (no --offline) and confirm the
464
+ // resulting CAR has the right number of blocks (full DAG minus one).
465
+ partialCarPath := filepath.Join(node.Dir, "partial.car")
466
+ require.NoError(t, node.IPFSDagExport(root, partialCarPath, "--local-only"))
467
+
468
+ // 300KiB --raw-leaves yields root + 2 leaves, so removing one leaf
469
+ // leaves 2 blocks. Asserting the exact count proves --offline was
470
+ // actually applied (without it, the export would either fetch the
471
+ // missing block or fail differently).
472
+ require.Equal(t, 2, countCARBlocks(t, partialCarPath))
473
+}
474
+
475
+// TestDagExportLocalOnlySkipsSubtree verifies that when a non-leaf block is
476
+// missing, --local-only skips the entire subtree under it, not just the
477
+// missing block. Uses a small chunk size to force a depth>1 DAG so removing
478
+// an intermediate prunes many descendant blocks.
479
+func TestDagExportLocalOnlySkipsSubtree(t *testing.T) {
480
+ t.Parallel()
481
+ node := harness.NewT(t).NewNode().Init().StartDaemon()
482
+ defer node.StopDaemon()
483
+
484
+ // chunker=size-256 + 64 KiB → 256 leaves; max-file-links=174 forces
485
+ // at least one intermediate dag-pb layer between root and leaves
486
+ // (256 > 174). Both values are pinned so the DAG shape (and the
487
+ // counts below) survives any change to Import.* defaults or profiles.
488
+ root := node.IPFSAddDeterministic("64KiB", "dag-export-local-only-subtree",
489
+ "--raw-leaves", "--chunker=size-256", "--max-file-links=174")
490
+ fullCarPath := filepath.Join(node.Dir, "full.car")
491
+ require.NoError(t, node.IPFSDagExport(root, fullCarPath))
492
+ fullCount := countCARBlocks(t, fullCarPath)
493
+ // 1 root + 2 intermediates (174 + 82 children) + 256 leaves = 259.
494
+ require.Equal(t, 259, fullCount, "expected root + 2 intermediates + 256 leaves, got %d", fullCount)
495
+
496
+ // Find the first intermediate ref: a non-leaf whose codec is dag-pb.
497
+ // "ipfs refs -r --unique" lists CIDs depth-first; the root's first
498
+ // child in a balanced UnixFS DAG with >174 leaves is an intermediate.
499
+ refs := dagRefs(node, root)
500
+ intermediate := refs[1]
501
+ intermediateChildren := dagRefs(node, intermediate)
502
+ require.Greater(t, len(intermediateChildren), 10,
503
+ "expected refs[1] to be a non-leaf with many children, got %d", len(intermediateChildren))
504
+
505
+ // Remove the intermediate. Its subtree blocks remain locally, but
506
+ // without the intermediate the walker cannot reach them, so they
507
+ // must be skipped along with it.
508
+ require.Equal(t, 0, node.RunIPFS("pin", "rm", root).ExitCode())
509
+ require.Equal(t, 0, node.RunIPFS("block", "rm", intermediate).ExitCode())
510
+
511
+ partialCarPath := filepath.Join(node.Dir, "partial.car")
512
+ require.NoError(t, node.IPFSDagExport(root, partialCarPath, "--local-only"))
513
+ partialCount := countCARBlocks(t, partialCarPath)
514
+
515
+ expectedDropped := len(intermediateChildren) // includes the intermediate itself
516
+ require.Equal(t, fullCount-expectedDropped, partialCount,
517
+ "removing intermediate %s should drop it and its %d descendants (full=%d, partial=%d)",
518
+ intermediate, expectedDropped-1, fullCount, partialCount)
519
+}
520
+
521
+// TestDagExportLocalOnlyConflictsWithOnline verifies that explicitly asking
522
+// for online mode together with --local-only is rejected, since the two
523
+// settings contradict each other.
524
+func TestDagExportLocalOnlyConflictsWithOnline(t *testing.T) {
525
+ t.Parallel()
526
+ node := harness.NewT(t).NewNode().Init().StartDaemon()
527
+ defer node.StopDaemon()
528
+
529
+ root := node.IPFSAddDeterministic("300KiB", "dag-export-local-only-online", "--raw-leaves")
530
+
531
+ res := node.RunIPFS("dag", "export", "--local-only", "--offline=false", root)
532
+ require.NotEqual(t, 0, res.ExitCode(), "dag export --local-only --offline=false should be rejected")
533
+ stderr := res.Stderr.String()
534
+ require.Contains(t, stderr, "--local-only")
535
+ require.Contains(t, stderr, "--offline")
536
+}
537
+
538
+// TestDagImportPartialCAR is the round-trip happy path: a partial CAR from
539
+// --local-only can be imported on a fresh node with default flags (the
540
+// IPFSDagImport harness helper passes --pin-roots=false). The helper also
541
+// confirms the root resolves offline on the receiver.
542
+func TestDagImportPartialCAR(t *testing.T) {
543
+ t.Parallel()
544
+ node := harness.NewT(t).NewNode().Init().StartDaemon()
545
+ defer node.StopDaemon()
546
+
547
+ root, _ := makePartialDAG(t, node, "dag-import-partial")
548
+
549
+ partialCarPath := filepath.Join(node.Dir, "partial.car")
550
+ require.NoError(t, node.IPFSDagExport(root, partialCarPath, "--local-only", "--offline"))
551
+
552
+ imp := harness.NewT(t).NewNode().Init().StartDaemon()
553
+ defer imp.StopDaemon()
554
+ partialCAR, err := os.Open(partialCarPath)
555
+ require.NoError(t, err)
556
+ defer partialCAR.Close()
557
+ require.NoError(t, imp.IPFSDagImport(partialCAR, root))
558
+}
559
+
560
+// TestDagImportLocalOnlyImpliesNoPin verifies that --local-only on its own
561
+// makes a partial-CAR import succeed: it implies --pin-roots=false so the
562
+// user does not have to pass both flags.
563
+func TestDagImportLocalOnlyImpliesNoPin(t *testing.T) {
564
+ t.Parallel()
565
+ node := harness.NewT(t).NewNode().Init().StartDaemon()
566
+ defer node.StopDaemon()
567
+
568
+ root, _ := makePartialDAG(t, node, "dag-import-local-only-implies")
569
+ partialCarPath := filepath.Join(node.Dir, "partial.car")
570
+ require.NoError(t, node.IPFSDagExport(root, partialCarPath, "--local-only", "--offline"))
571
+
572
+ imp := harness.NewT(t).NewNode().Init().StartDaemon()
573
+ defer imp.StopDaemon()
574
+ partialCAR, err := os.Open(partialCarPath)
575
+ require.NoError(t, err)
576
+ defer partialCAR.Close()
577
+
578
+ // Import with only --local-only (no --pin-roots=false). Should
579
+ // succeed because --local-only implies --pin-roots=false, and the
580
+ // receiver must not attempt to pin (pin would fail on a partial DAG).
581
+ res := imp.Runner.Run(harness.RunRequest{
582
+ Path: imp.IPFSBin,
583
+ Args: []string{"dag", "import", "--local-only"},
584
+ CmdOpts: []harness.CmdOpt{harness.RunWithStdin(partialCAR)},
585
+ })
586
+ require.Equal(t, 0, res.ExitCode(),
587
+ "dag import --local-only on a partial CAR should succeed; stderr: %s", res.Stderr.String())
588
+ require.NotContains(t, res.Stdout.String(), "Pinned root",
589
+ "import must not pin when --local-only is set")
590
+}
591
+
592
+// TestDagImportLocalOnlyPinRootsConflict verifies that --local-only is
593
+// rejected when combined with an explicit --pin-roots=true. The two are
594
+// mutually exclusive: --local-only is for partial CARs (no full DAG to pin).
595
+func TestDagImportLocalOnlyPinRootsConflict(t *testing.T) {
596
+ t.Parallel()
597
+ node := harness.NewT(t).NewNode().Init().StartDaemon()
598
+ defer node.StopDaemon()
599
+
600
+ r, err := os.Open(fixtureFile)
601
+ require.NoError(t, err)
602
+ defer r.Close()
603
+
604
+ res := node.Runner.Run(harness.RunRequest{
605
+ Path: node.IPFSBin,
606
+ Args: []string{"dag", "import", "--local-only", "--pin-roots=true"},
607
+ CmdOpts: []harness.CmdOpt{harness.RunWithStdin(r)},
608
+ })
609
+
610
+ require.NotEqual(t, 0, res.ExitCode())
611
+ stderr := res.Stderr.String()
612
+ require.Contains(t, stderr, "--local-only")
613
+ require.Contains(t, stderr, "--pin-roots")
614
+}
test/cli/harness/ipfs.go
+5
-3
@@ -162,17 +162,19 @@ func (n *Node) IPFSDagImport(content io.Reader, cid string, args ...string) erro
162
}
163
164
// IPFSDagExport exports a DAG rooted at cid to a CAR file at carPath.
165
-func (n *Node) IPFSDagExport(cid string, carPath string) error {
166
- log.Debugf("node %d dag export of %s to %q", n.ID, cid, carPath)
165
+func (n *Node) IPFSDagExport(cid string, carPath string, args ...string) error {
166
+ log.Debugf("node %d dag export of %s to %q with args: %v", n.ID, cid, carPath, args)
167
car, err := os.Create(carPath)
168
if err != nil {
169
return err
170
}
171
defer car.Close()
172
173
+ fullArgs := append([]string{"dag", "export"}, args...)
174
+ fullArgs = append(fullArgs, cid)
175
res := n.Runner.MustRun(RunRequest{
176
Path: n.IPFSBin,
175
- Args: []string{"dag", "export", cid},
177
+ Args: fullArgs,
178
CmdOpts: []CmdOpt{RunWithStdout(car)},
179
})
180
return res.Err
test/dependencies/go.mod
+1
-1
@@ -149,7 +149,7 @@ require (
149
github.com/ipfs/go-metrics-interface v0.3.0 // indirect
150
github.com/ipfs/go-unixfsnode v1.10.4 // indirect
151
github.com/ipfs/kubo v0.31.0 // indirect
152
- github.com/ipld/go-car/v2 v2.16.0 // indirect
152
+ github.com/ipld/go-car/v2 v2.16.1-0.20260428045700-c4b9f366f20c // indirect
153
github.com/ipld/go-codec-dagpb v1.7.0 // indirect
154
github.com/ipld/go-ipld-prime v0.23.0 // indirect
155
github.com/ipshipyard/p2p-forge v0.8.0 // indirect
test/dependencies/go.sum
+2
-2
@@ -500,8 +500,8 @@ github.com/ipfs/iptb v1.4.1 h1:faXd3TKGPswbHyZecqqg6UfbES7RDjTKQb+6VFPKDUo=
500
github.com/ipfs/iptb v1.4.1/go.mod h1:nTsBMtVYFEu0FjC5DgrErnABm3OG9ruXkFXGJoTV5OA=
501
github.com/ipfs/iptb-plugins v0.5.1 h1:11PNTNEt2+SFxjUcO5qpyCTXqDj6T8Tx9pU/G4ytCIQ=
502
github.com/ipfs/iptb-plugins v0.5.1/go.mod h1:mscJAjRnu4g16QK6oUBn9RGpcp8ueJmLfmPxIG/At78=
503
-github.com/ipld/go-car/v2 v2.16.0 h1:LWe0vmN/QcQmUU4tr34W5Nv5mNraW+G6jfN2s+ndBco=
504
-github.com/ipld/go-car/v2 v2.16.0/go.mod h1:RqFGWN9ifcXVmCrTAVnfnxiWZk1+jIx67SYhenlmL34=
503
+github.com/ipld/go-car/v2 v2.16.1-0.20260428045700-c4b9f366f20c h1:ZFONxHSj6bzzB9eKIu+yS2AazTJe7j9FPesfy4sZSE0=
504
+github.com/ipld/go-car/v2 v2.16.1-0.20260428045700-c4b9f366f20c/go.mod h1:/4HY8tFZ1q42Mw54ILLPQfjkUqMJxFKqY1yMDKHlYko=
505
github.com/ipld/go-codec-dagpb v1.7.0 h1:hpuvQjCSVSLnTnHXn+QAMR0mLmb1gA6wl10LExo2Ts0=
506
github.com/ipld/go-codec-dagpb v1.7.0/go.mod h1:rD3Zg+zub9ZnxcLwfol/OTQRVjaLzXypgy4UqHQvilM=
507
github.com/ipld/go-ipld-prime v0.23.0 h1:csqdPZH60BsTC+AZrv7fpa27v+09I/oTqyHYYYE27eE=