feat(config): add Import.* for CID Profiles from IPIP-499 (#11148)
* feat(config): Import.* and unixfs-v1-2025 profile implements IPIP-499: add config options for controlling UnixFS DAG determinism and introduces `unixfs-v1-2025` and `unixfs-v0-2015` profiles for cross-implementation CID reproducibility. changes: - add Import.* fields: HAMTDirectorySizeEstimation, SymlinkMode, DAGLayout, IncludeEmptyDirectories, IncludeHidden - add validation for all Import.* config values - add unixfs-v1-2025 profile (recommended for new data) - add unixfs-v0-2015 profile (alias: legacy-cid-v0) - remove deprecated test-cid-v1 and test-cid-v1-wide profiles - wire Import.HAMTSizeEstimationMode() to boxo globals - update go.mod to use boxo with SizeEstimationMode support ref: https://specs.ipfs.tech/ipips/ipip-0499/ * feat(add): add --dereference-symlinks, --empty-dirs, --hidden CLI flags add CLI flags for controlling file collection behavior during ipfs add: - `--dereference-symlinks`: recursively resolve symlinks to their target content (replaces deprecated --dereference-args which only worked on CLI arguments). wired through go-ipfs-cmds to boxo's SerialFileOptions. - `--empty-dirs` / `-E`: include empty directories (default: true) - `--hidden` / `-H`: include hidden files (default: false) these flags are CLI-only and not wired to Import.* config options because go-ipfs-cmds library handles input file filtering before the directory tree is passed to kubo. removed unused Import.UnixFSSymlinkMode config option that was defined but never actually read by the CLI. also: - wire --trickle to Import.UnixFSDAGLayout config default - update go-ipfs-cmds to v0.15.1-0.20260117043932-17687e216294 - add SYMLINK HANDLING section to ipfs add help text - add CLI tests for all three flags ref: https://github.com/ipfs/specs/pull/499 * test(add): add CID profile tests and wire SizeEstimationMode add comprehensive test suite for UnixFS CID determinism per IPIP-499: - verify exact HAMT threshold boundary for both estimation modes: - v0-2015 (links): sum(name_len + cid_len) == 262144 - v1-2025 (block): serialized block size == 262144 - verify HAMT triggers at threshold + 1 byte for both profiles - add all deterministic CIDs for cross-implementation testing also wires SizeEstimationMode through CLI/API, allowing Import.UnixFSHAMTSizeEstimation config to take effect. bumps boxo to ipfs/boxo@6707376 which aligns HAMT threshold with JS implementation (uses > instead of >=), fixing CID determinism at the exact 256 KiB boundary. * feat(add): --dereference-symlinks now resolves all symlinks Previously, resolving symlinks required two flags: - --dereference-args: resolved symlinks passed as CLI arguments - --dereference-symlinks: resolved symlinks inside directories Now --dereference-symlinks handles both cases. Users only need one flag to fully dereference symlinks when adding files to IPFS. The deprecated --dereference-args still works for backwards compatibility but is no longer necessary. * chore: update boxo and improve changelog - update boxo to ebdaf07c (nil filter fix, thread-safety docs) - simplify changelog for IPIP-499 section - shorten test names, move context to comments * chore: update boxo to 5cf22196 * chore: apply suggestions from code review Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> * test(add): verify balanced DAG layout produces uniform leaf depth add test that confirms kubo uses balanced layout (all leaves at same depth) rather than balanced-packed (varying depths). creates 45MiB file to trigger multi-level DAG and walks it to verify leaf depth uniformity. includes trickle subtest to validate test logic can detect varying depths. supports CAR export via DAG_LAYOUT_CAR_OUTPUT env var for test vectors. * chore(deps): update boxo to 6141039ad8ef switches to https://github.com/ipfs/boxo/pull/1088/commits/6141039ad8ef098c3b65db8b2d1aeb3c16727c6c changes since 5cf22196ad0b: - refactor(unixfs): use arithmetic for exact block size calculation - refactor(unixfs): unify size tracking and make SizeEstimationMode immutable - feat(unixfs): optimize SizeEstimationBlock and add mode/mtime tests also clarifies that directory sharding globals affect both `ipfs add` and MFS. * test(cli): improve HAMT threshold tests with exact +1 byte verification - add UnixFSDataType() helper to directly check UnixFS type via protobuf - refactor threshold tests to use exact +1 byte calculations instead of +1 file - verify directory type directly (ft.TDirectory vs ft.THAMTShard) instead of inferring from link count - clean up helper function signatures by removing unused cidLength parameter * test(cli): consolidate profile tests into cid_profiles_test.go remove duplicate profile threshold tests from add_test.go since they are fully covered by the data-driven tests in cid_profiles_test.go. changes: - improve test names to describe what threshold is being tested - add inline documentation explaining each test's purpose - add byte-precise helper IPFSAddDeterministicBytes for threshold tests - remove ~200 lines of duplicated test code from add_test.go - keep non-profile tests (pinning, symlinks, hidden files) in add_test.go * chore: update to rebased boxo and go-ipfs-cmds PRs * docs: add HAMT threshold fix details to changelog * feat(mfs): use Import config for CID version and hash function make MFS commands (files cp, files write, files mkdir, files chcid) respect Import.CidVersion and Import.HashFunction config settings when CLI options are not explicitly provided. also add tests for: - files write respects Import.UnixFSRawLeaves=true - single-block file: files write produces same CID as ipfs add - updated comments clarifying CID parity with ipfs add * feat(files): wire Import.UnixFSChunker and UnixFSDirectoryMaxLinks to MFS `ipfs files` commands now respect these Import.* config options: - UnixFSChunker: configures chunk size for `files write` - UnixFSDirectoryMaxLinks: triggers HAMT sharding in `files mkdir` - UnixFSHAMTDirectorySizeEstimation: controls size estimation mode previously, MFS used hardcoded defaults ignoring user config. changes: - config/import.go: add UnixFSSplitterFunc() returning chunk.SplitterGen - core/node/core.go: pass chunker, maxLinks, sizeEstimationMode to mfs.NewRoot() via new boxo RootOption API - core/commands/files.go: pass maxLinks and sizeEstimationMode to mfs.Mkdir() and ensureContainingDirectoryExists(); document that UnixFSFileMaxLinks doesn't apply to files write (trickle DAG limitation) - test/cli/files_test.go: add tests for UnixFSDirectoryMaxLinks and UnixFSChunker, including CID parity test with `ipfs add --trickle` related: boxo@54e044f1b265 * feat(files): wire Import.UnixFSHAMTDirectoryMaxFanout and UnixFSHAMTDirectorySizeThreshold wire remaining HAMT config options to MFS root: - Import.UnixFSHAMTDirectoryMaxFanout via mfs.WithMaxHAMTFanout - Import.UnixFSHAMTDirectorySizeThreshold via mfs.WithHAMTShardingSize add CLI tests: - files mkdir respects Import.UnixFSHAMTDirectoryMaxFanout - files mkdir respects Import.UnixFSHAMTDirectorySizeThreshold - config change takes effect after daemon restart add UnixFSHAMTFanout() helper to test harness update boxo to ac97424d99ab90e097fc7c36f285988b596b6f05 * fix(mfs): single-block files in CIDv1 dirs now produce raw CIDs problem: `ipfs files write` in CIDv1 directories wrapped single-block files in dag-pb even when raw-leaves was enabled, producing different CIDs than `ipfs add --raw-leaves` for the same content. fix: boxo now collapses single-block ProtoNode wrappers (with no metadata) to RawNode in DagModifier.GetNode(). files with mtime/mode stay as dag-pb since raw blocks cannot store UnixFS metadata. also fixes sparse file writes where writing past EOF would lose data because expandSparse didn't update the internal node pointer. updates boxo to v0.36.1-0.20260203003133-7884ae23aaff updates t0250-files-api.sh test hashes to match new behavior * chore(test): use Go 1.22+ range-over-int syntax * chore: update boxo to c6829fe26860 - fix typo in files write help text - update boxo with CI fixes (gofumpt, race condition in test) * chore: update go-ipfs-cmds to 192ec9d15c1f includes binary content types fix: gzip, zip, vnd.ipld.car, vnd.ipld.raw, vnd.ipfs.ipns-record * chore: update boxo to 0a22cde9225c includes refactor of maxLinks check in addLinkChild (review feedback). * ci: fix helia-interop and improve caching skip '@helia/mfs - should have the same CID after creating a file' test until helia implements IPIP-499 (tracking: https://github.com/ipfs/helia/issues/941) the test fails because kubo now collapses single-block files to raw CIDs while helia explicitly uses reduceSingleLeafToSelf: false changes: - run aegir directly instead of helia-interop binary (binary ignores --grep flags) - cache node_modules keyed by @helia/interop version from npm registry - skip npm install on cache hit (matches ipfs-webui caching pattern) * chore: update boxo to 1e30b954 includes latest upstream changes from boxo main * chore: update go-ipfs-cmds to 1b2a641ed6f6 * chore: update boxo to f188f79fd412 switches to boxo@main after merging https://github.com/ipfs/boxo/pull/1088 * chore: update go-ipfs-cmds to af9bcbaf5709 switches to go-ipfs-cmds@master after merging https://github.com/ipfs/go-ipfs-cmds/pull/315 --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
Marcin Rataj committed
Feb 4, 2026 at 22:03 UTC
67c89bbd7e13ea6778fb1c17c4373320b7b878e2
28 files changed
+2311
-447
.github/workflows/interop.yml
+32
-8
@@ -71,18 +71,42 @@ jobs:
71
name: kubo
72
path: cmd/ipfs
73
- run: chmod +x cmd/ipfs/ipfs
74
- - run: echo "dir=$(npm config get cache)" >> $GITHUB_OUTPUT
75
- id: npm-cache-dir
76
- - uses: actions/cache@v5
77
- with:
78
- path: ${{ steps.npm-cache-dir.outputs.dir }}
79
- key: ${{ runner.os }}-${{ github.job }}-helia-${{ hashFiles('**/package-lock.json') }}
80
- restore-keys: ${{ runner.os }}-${{ github.job }}-helia-
74
- run: sudo apt update
75
- run: sudo apt install -y libxkbcommon0 libxdamage1 libgbm1 libpango-1.0-0 libcairo2 # dependencies for playwright
83
- - run: npx --package @helia/interop helia-interop
76
+ # Cache node_modules based on latest @helia/interop version from npm registry.
77
+ # This ensures we always test against the latest release while still benefiting
78
+ # from caching when the version hasn't changed.
79
+ - name: Get latest @helia/interop version
80
+ id: helia-version
81
+ run: echo "version=$(npm view @helia/interop version)" >> $GITHUB_OUTPUT
82
+ - name: Cache helia-interop node_modules
83
+ uses: actions/cache@v5
84
+ id: helia-cache
85
+ with:
86
+ path: node_modules
87
+ key: ${{ runner.os }}-helia-interop-${{ steps.helia-version.outputs.version }}
88
+ - name: Install @helia/interop
89
+ if: steps.helia-cache.outputs.cache-hit != 'true'
90
+ run: npm install @helia/interop
91
+ # TODO(IPIP-499): Remove --grep --invert workaround once helia implements IPIP-499
92
+ # Tracking issue: https://github.com/ipfs/helia/issues/941
93
+ #
94
+ # PROVISIONAL HACK: Skip '@helia/mfs - should have the same CID after
95
+ # creating a file' test due to IPIP-499 changes in kubo.
96
+ #
97
+ # WHY IT FAILS: The test creates a 5-byte file in MFS on both kubo and helia,
98
+ # then compares the root directory CID. With kubo PR #11148, `ipfs files write`
99
+ # now produces raw CIDs for single-block files (matching `ipfs add --raw-leaves`),
100
+ # while helia uses `reduceSingleLeafToSelf: false` which keeps the dag-pb wrapper.
101
+ # Different file CIDs lead to different directory CIDs.
102
+ #
103
+ # We run aegir directly (instead of helia-interop binary) because only aegir
104
+ # supports the --grep/--invert flags needed to exclude specific tests.
105
+ - name: Run helia-interop tests (excluding IPIP-499 incompatible test)
106
+ run: npx aegir test -t node --bail -- --grep 'should have the same CID after creating a file' --invert
107
env:
108
KUBO_BINARY: ${{ github.workspace }}/cmd/ipfs/ipfs
109
+ working-directory: node_modules/@helia/interop
110
ipfs-webui:
111
needs: [interop-prep]
112
runs-on: ${{ fromJSON(github.repository == 'ipfs/kubo' && '["self-hosted", "linux", "x64", "2xlarge"]' || '"ubuntu-latest"') }}
config/import.go
+95
-17
@@ -2,11 +2,13 @@ package config
2
3
import (
4
"fmt"
5
+ "io"
6
"strconv"
7
"strings"
8
9
+ chunk "github.com/ipfs/boxo/chunker"
10
"github.com/ipfs/boxo/ipld/unixfs/importer/helpers"
9
- "github.com/ipfs/boxo/ipld/unixfs/io"
11
+ uio "github.com/ipfs/boxo/ipld/unixfs/io"
12
"github.com/ipfs/boxo/verifcid"
13
mh "github.com/multiformats/go-multihash"
14
)
@@ -29,29 +31,44 @@ const (
31
// write-batch. The total size of the batch is limited by
32
// BatchMaxnodes and BatchMaxSize.
33
DefaultBatchMaxSize = 100 << 20 // 20MiB
34
+
35
+ // HAMTSizeEstimation values for Import.UnixFSHAMTDirectorySizeEstimation
36
+ HAMTSizeEstimationLinks = "links" // legacy: estimate using link names + CID byte lengths (default)
37
+ HAMTSizeEstimationBlock = "block" // full serialized dag-pb block size
38
+ HAMTSizeEstimationDisabled = "disabled" // disable HAMT sharding entirely
39
+
40
+ // DAGLayout values for Import.UnixFSDAGLayout
41
+ DAGLayoutBalanced = "balanced" // balanced DAG layout (default)
42
+ DAGLayoutTrickle = "trickle" // trickle DAG layout
43
+
44
+ DefaultUnixFSHAMTDirectorySizeEstimation = HAMTSizeEstimationLinks // legacy behavior
45
+ DefaultUnixFSDAGLayout = DAGLayoutBalanced // balanced DAG layout
46
+ DefaultUnixFSIncludeEmptyDirs = true // include empty directories
47
)
48
49
var (
50
DefaultUnixFSFileMaxLinks = int64(helpers.DefaultLinksPerBlock)
51
DefaultUnixFSDirectoryMaxLinks = int64(0)
37
- DefaultUnixFSHAMTDirectoryMaxFanout = int64(io.DefaultShardWidth)
52
+ DefaultUnixFSHAMTDirectoryMaxFanout = int64(uio.DefaultShardWidth)
53
)
54
55
// Import configures the default options for ingesting data. This affects commands
56
// that ingest data, such as 'ipfs add', 'ipfs dag put, 'ipfs block put', 'ipfs files write'.
57
type Import struct {
43
- CidVersion OptionalInteger
44
- UnixFSRawLeaves Flag
45
- UnixFSChunker OptionalString
46
- HashFunction OptionalString
47
- UnixFSFileMaxLinks OptionalInteger
48
- UnixFSDirectoryMaxLinks OptionalInteger
49
- UnixFSHAMTDirectoryMaxFanout OptionalInteger
50
- UnixFSHAMTDirectorySizeThreshold OptionalBytes
51
- BatchMaxNodes OptionalInteger
52
- BatchMaxSize OptionalInteger
53
- FastProvideRoot Flag
54
- FastProvideWait Flag
58
+ CidVersion OptionalInteger
59
+ UnixFSRawLeaves Flag
60
+ UnixFSChunker OptionalString
61
+ HashFunction OptionalString
62
+ UnixFSFileMaxLinks OptionalInteger
63
+ UnixFSDirectoryMaxLinks OptionalInteger
64
+ UnixFSHAMTDirectoryMaxFanout OptionalInteger
65
+ UnixFSHAMTDirectorySizeThreshold OptionalBytes
66
+ UnixFSHAMTDirectorySizeEstimation OptionalString // "links", "block", or "disabled"
67
+ UnixFSDAGLayout OptionalString // "balanced" or "trickle"
68
+ BatchMaxNodes OptionalInteger
69
+ BatchMaxSize OptionalInteger
70
+ FastProvideRoot Flag
71
+ FastProvideWait Flag
72
}
73
74
// ValidateImportConfig validates the Import configuration according to UnixFS spec requirements.
@@ -129,6 +146,30 @@ func ValidateImportConfig(cfg *Import) error {
146
}
147
}
148
149
+ // Validate UnixFSHAMTDirectorySizeEstimation
150
+ if !cfg.UnixFSHAMTDirectorySizeEstimation.IsDefault() {
151
+ est := cfg.UnixFSHAMTDirectorySizeEstimation.WithDefault(DefaultUnixFSHAMTDirectorySizeEstimation)
152
+ switch est {
153
+ case HAMTSizeEstimationLinks, HAMTSizeEstimationBlock, HAMTSizeEstimationDisabled:
154
+ // valid
155
+ default:
156
+ return fmt.Errorf("Import.UnixFSHAMTDirectorySizeEstimation must be %q, %q, or %q, got %q",
157
+ HAMTSizeEstimationLinks, HAMTSizeEstimationBlock, HAMTSizeEstimationDisabled, est)
158
+ }
159
+ }
160
+
161
+ // Validate UnixFSDAGLayout
162
+ if !cfg.UnixFSDAGLayout.IsDefault() {
163
+ layout := cfg.UnixFSDAGLayout.WithDefault(DefaultUnixFSDAGLayout)
164
+ switch layout {
165
+ case DAGLayoutBalanced, DAGLayoutTrickle:
166
+ // valid
167
+ default:
168
+ return fmt.Errorf("Import.UnixFSDAGLayout must be %q or %q, got %q",
169
+ DAGLayoutBalanced, DAGLayoutTrickle, layout)
170
+ }
171
+ }
172
+
173
return nil
174
}
175
@@ -144,8 +185,7 @@ func isValidChunker(chunker string) bool {
185
}
186
187
// Check for size-<bytes> format
147
- if strings.HasPrefix(chunker, "size-") {
148
- sizeStr := strings.TrimPrefix(chunker, "size-")
188
+ if sizeStr, ok := strings.CutPrefix(chunker, "size-"); ok {
189
if sizeStr == "" {
190
return false
191
}
@@ -167,7 +207,7 @@ func isValidChunker(chunker string) bool {
207
208
// Parse and validate min, avg, max values
209
values := make([]int, 3)
170
- for i := 0; i < 3; i++ {
210
+ for i := range 3 {
211
val, err := strconv.Atoi(parts[i+1])
212
if err != nil {
213
return false
@@ -182,3 +222,41 @@ func isValidChunker(chunker string) bool {
222
223
return false
224
}
225
+
226
+// HAMTSizeEstimationMode returns the boxo SizeEstimationMode based on the config value.
227
+func (i *Import) HAMTSizeEstimationMode() uio.SizeEstimationMode {
228
+ switch i.UnixFSHAMTDirectorySizeEstimation.WithDefault(DefaultUnixFSHAMTDirectorySizeEstimation) {
229
+ case HAMTSizeEstimationLinks:
230
+ return uio.SizeEstimationLinks
231
+ case HAMTSizeEstimationBlock:
232
+ return uio.SizeEstimationBlock
233
+ case HAMTSizeEstimationDisabled:
234
+ return uio.SizeEstimationDisabled
235
+ default:
236
+ return uio.SizeEstimationLinks
237
+ }
238
+}
239
+
240
+// UnixFSSplitterFunc returns a SplitterGen function based on Import.UnixFSChunker.
241
+// The returned function creates a Splitter for the configured chunking strategy.
242
+// The chunker string is parsed once when this method is called, not on each use.
243
+func (i *Import) UnixFSSplitterFunc() chunk.SplitterGen {
244
+ chunkerStr := i.UnixFSChunker.WithDefault(DefaultUnixFSChunker)
245
+
246
+ // Parse size-based chunker (most common case) and return optimized generator
247
+ if sizeStr, ok := strings.CutPrefix(chunkerStr, "size-"); ok {
248
+ if size, err := strconv.ParseInt(sizeStr, 10, 64); err == nil && size > 0 {
249
+ return chunk.SizeSplitterGen(size)
250
+ }
251
+ }
252
+
253
+ // For other chunker types (rabin, buzhash) or invalid config,
254
+ // fall back to parsing per-use (these are rare cases)
255
+ return func(r io.Reader) chunk.Splitter {
256
+ s, err := chunk.FromString(r, chunkerStr)
257
+ if err != nil {
258
+ return chunk.DefaultSplitter(r)
259
+ }
260
+ return s
261
+ }
262
+}
config/import_test.go
+102
@@ -4,6 +4,7 @@ import (
4
"strings"
5
"testing"
6
7
+ "github.com/ipfs/boxo/ipld/unixfs/io"
8
mh "github.com/multiformats/go-multihash"
9
)
10
@@ -406,3 +407,104 @@ func TestIsPowerOfTwo(t *testing.T) {
407
})
408
}
409
}
410
+
411
+func TestValidateImportConfig_HAMTSizeEstimation(t *testing.T) {
412
+ tests := []struct {
413
+ name string
414
+ value string
415
+ wantErr bool
416
+ errMsg string
417
+ }{
418
+ {name: "valid links", value: HAMTSizeEstimationLinks, wantErr: false},
419
+ {name: "valid block", value: HAMTSizeEstimationBlock, wantErr: false},
420
+ {name: "valid disabled", value: HAMTSizeEstimationDisabled, wantErr: false},
421
+ {name: "invalid unknown", value: "unknown", wantErr: true, errMsg: "must be"},
422
+ {name: "invalid empty", value: "", wantErr: true, errMsg: "must be"},
423
+ {name: "invalid typo", value: "link", wantErr: true, errMsg: "must be"},
424
+ }
425
+
426
+ for _, tt := range tests {
427
+ t.Run(tt.name, func(t *testing.T) {
428
+ cfg := &Import{
429
+ UnixFSHAMTDirectorySizeEstimation: *NewOptionalString(tt.value),
430
+ }
431
+
432
+ err := ValidateImportConfig(cfg)
433
+
434
+ if tt.wantErr {
435
+ if err == nil {
436
+ t.Errorf("expected error for value=%q, got nil", tt.value)
437
+ } else if tt.errMsg != "" && !strings.Contains(err.Error(), tt.errMsg) {
438
+ t.Errorf("error = %v, want error containing %q", err, tt.errMsg)
439
+ }
440
+ } else {
441
+ if err != nil {
442
+ t.Errorf("unexpected error for value=%q: %v", tt.value, err)
443
+ }
444
+ }
445
+ })
446
+ }
447
+}
448
+
449
+func TestValidateImportConfig_DAGLayout(t *testing.T) {
450
+ tests := []struct {
451
+ name string
452
+ value string
453
+ wantErr bool
454
+ errMsg string
455
+ }{
456
+ {name: "valid balanced", value: DAGLayoutBalanced, wantErr: false},
457
+ {name: "valid trickle", value: DAGLayoutTrickle, wantErr: false},
458
+ {name: "invalid unknown", value: "unknown", wantErr: true, errMsg: "must be"},
459
+ {name: "invalid empty", value: "", wantErr: true, errMsg: "must be"},
460
+ {name: "invalid flat", value: "flat", wantErr: true, errMsg: "must be"},
461
+ }
462
+
463
+ for _, tt := range tests {
464
+ t.Run(tt.name, func(t *testing.T) {
465
+ cfg := &Import{
466
+ UnixFSDAGLayout: *NewOptionalString(tt.value),
467
+ }
468
+
469
+ err := ValidateImportConfig(cfg)
470
+
471
+ if tt.wantErr {
472
+ if err == nil {
473
+ t.Errorf("expected error for value=%q, got nil", tt.value)
474
+ } else if tt.errMsg != "" && !strings.Contains(err.Error(), tt.errMsg) {
475
+ t.Errorf("error = %v, want error containing %q", err, tt.errMsg)
476
+ }
477
+ } else {
478
+ if err != nil {
479
+ t.Errorf("unexpected error for value=%q: %v", tt.value, err)
480
+ }
481
+ }
482
+ })
483
+ }
484
+}
485
+
486
+func TestImport_HAMTSizeEstimationMode(t *testing.T) {
487
+ tests := []struct {
488
+ cfg string
489
+ want io.SizeEstimationMode
490
+ }{
491
+ {HAMTSizeEstimationLinks, io.SizeEstimationLinks},
492
+ {HAMTSizeEstimationBlock, io.SizeEstimationBlock},
493
+ {HAMTSizeEstimationDisabled, io.SizeEstimationDisabled},
494
+ {"", io.SizeEstimationLinks}, // default (unset returns default)
495
+ {"unknown", io.SizeEstimationLinks}, // fallback to default
496
+ }
497
+
498
+ for _, tt := range tests {
499
+ t.Run(tt.cfg, func(t *testing.T) {
500
+ var imp Import
501
+ if tt.cfg != "" {
502
+ imp.UnixFSHAMTDirectorySizeEstimation = *NewOptionalString(tt.cfg)
503
+ }
504
+ got := imp.HAMTSizeEstimationMode()
505
+ if got != tt.want {
506
+ t.Errorf("Import.HAMTSizeEstimationMode() with %q = %v, want %v", tt.cfg, got, tt.want)
507
+ }
508
+ })
509
+ }
510
+}
config/profile.go
+33
-30
@@ -312,45 +312,33 @@ fetching may be degraded.
312
return nil
313
},
314
},
315
+ "unixfs-v0-2015": {
316
+ Description: `Legacy UnixFS import profile for backward-compatible CID generation.
317
+Produces CIDv0 with no raw leaves, sha2-256, 256 KiB chunks, and
318
+link-based HAMT size estimation. Use only when legacy CIDs are required.
319
+See https://github.com/ipfs/specs/pull/499. Alias: legacy-cid-v0`,
320
+ Transform: applyUnixFSv02015,
321
+ },
322
"legacy-cid-v0": {
316
- Description: `Makes UnixFS import produce legacy CIDv0 with no raw leaves, sha2-256 and 256 KiB chunks. This is likely the least optimal preset, use only if legacy behavior is required.`,
317
- Transform: func(c *Config) error {
318
- c.Import.CidVersion = *NewOptionalInteger(0)
319
- c.Import.UnixFSRawLeaves = False
320
- c.Import.UnixFSChunker = *NewOptionalString("size-262144")
321
- c.Import.HashFunction = *NewOptionalString("sha2-256")
322
- c.Import.UnixFSFileMaxLinks = *NewOptionalInteger(174)
323
- c.Import.UnixFSDirectoryMaxLinks = *NewOptionalInteger(0)
324
- c.Import.UnixFSHAMTDirectoryMaxFanout = *NewOptionalInteger(256)
325
- c.Import.UnixFSHAMTDirectorySizeThreshold = *NewOptionalBytes("256KiB")
326
- return nil
327
- },
323
+ Description: `Alias for unixfs-v0-2015 profile.`,
324
+ Transform: applyUnixFSv02015,
325
},
329
- "test-cid-v1": {
330
- Description: `Makes UnixFS import produce CIDv1 with raw leaves, sha2-256 and 1 MiB chunks (max 174 links per file, 256 per HAMT node, switch dir to HAMT above 256KiB).`,
326
+ "unixfs-v1-2025": {
327
+ Description: `Recommended UnixFS import profile for cross-implementation CID determinism.
328
+Uses CIDv1, raw leaves, sha2-256, 1 MiB chunks, 1024 links per file node,
329
+256 HAMT fanout, and block-based size estimation for HAMT threshold.
330
+See https://github.com/ipfs/specs/pull/499`,
331
Transform: func(c *Config) error {
332
c.Import.CidVersion = *NewOptionalInteger(1)
333
c.Import.UnixFSRawLeaves = True
334
- c.Import.UnixFSChunker = *NewOptionalString("size-1048576")
334
+ c.Import.UnixFSChunker = *NewOptionalString("size-1048576") // 1 MiB
335
c.Import.HashFunction = *NewOptionalString("sha2-256")
336
- c.Import.UnixFSFileMaxLinks = *NewOptionalInteger(174)
336
+ c.Import.UnixFSFileMaxLinks = *NewOptionalInteger(1024)
337
c.Import.UnixFSDirectoryMaxLinks = *NewOptionalInteger(0)
338
c.Import.UnixFSHAMTDirectoryMaxFanout = *NewOptionalInteger(256)
339
c.Import.UnixFSHAMTDirectorySizeThreshold = *NewOptionalBytes("256KiB")
340
- return nil
341
- },
342
- },
343
- "test-cid-v1-wide": {
344
- Description: `Makes UnixFS import produce CIDv1 with raw leaves, sha2-256 and 1MiB chunks and wider file DAGs (max 1024 links per every node type, switch dir to HAMT above 1MiB).`,
345
- Transform: func(c *Config) error {
346
- c.Import.CidVersion = *NewOptionalInteger(1)
347
- c.Import.UnixFSRawLeaves = True
348
- c.Import.UnixFSChunker = *NewOptionalString("size-1048576") // 1MiB
349
- c.Import.HashFunction = *NewOptionalString("sha2-256")
350
- c.Import.UnixFSFileMaxLinks = *NewOptionalInteger(1024)
351
- c.Import.UnixFSDirectoryMaxLinks = *NewOptionalInteger(0) // no limit here, use size-based Import.UnixFSHAMTDirectorySizeThreshold instead
352
- c.Import.UnixFSHAMTDirectoryMaxFanout = *NewOptionalInteger(1024)
353
- c.Import.UnixFSHAMTDirectorySizeThreshold = *NewOptionalBytes("1MiB") // 1MiB
340
+ c.Import.UnixFSHAMTDirectorySizeEstimation = *NewOptionalString(HAMTSizeEstimationBlock)
341
+ c.Import.UnixFSDAGLayout = *NewOptionalString(DAGLayoutBalanced)
342
return nil
343
},
344
},
@@ -435,3 +423,18 @@ func mapKeys(m map[string]struct{}) []string {
423
}
424
return out
425
}
426
+
427
+// applyUnixFSv02015 applies the legacy UnixFS v0 (2015) import settings.
428
+func applyUnixFSv02015(c *Config) error {
429
+ c.Import.CidVersion = *NewOptionalInteger(0)
430
+ c.Import.UnixFSRawLeaves = False
431
+ c.Import.UnixFSChunker = *NewOptionalString("size-262144") // 256 KiB
432
+ c.Import.HashFunction = *NewOptionalString("sha2-256")
433
+ c.Import.UnixFSFileMaxLinks = *NewOptionalInteger(174)
434
+ c.Import.UnixFSDirectoryMaxLinks = *NewOptionalInteger(0)
435
+ c.Import.UnixFSHAMTDirectoryMaxFanout = *NewOptionalInteger(256)
436
+ c.Import.UnixFSHAMTDirectorySizeThreshold = *NewOptionalBytes("256KiB")
437
+ c.Import.UnixFSHAMTDirectorySizeEstimation = *NewOptionalString(HAMTSizeEstimationLinks)
438
+ c.Import.UnixFSDAGLayout = *NewOptionalString(DAGLayoutBalanced)
439
+ return nil
440
+}
core/commands/add.go
+38
-2
@@ -15,6 +15,7 @@ import (
15
16
"github.com/cheggaaa/pb"
17
"github.com/ipfs/boxo/files"
18
+ uio "github.com/ipfs/boxo/ipld/unixfs/io"
19
mfs "github.com/ipfs/boxo/mfs"
20
"github.com/ipfs/boxo/path"
21
"github.com/ipfs/boxo/verifcid"
@@ -68,6 +69,7 @@ const (
69
mtimeNsecsOptionName = "mtime-nsecs"
70
fastProvideRootOptionName = "fast-provide-root"
71
fastProvideWaitOptionName = "fast-provide-wait"
72
+ emptyDirsOptionName = "empty-dirs"
73
)
74
75
const (
@@ -147,6 +149,18 @@ to find it in the future:
149
See 'ipfs files --help' to learn more about using MFS
150
for keeping track of added files and directories.
151
152
+SYMLINK HANDLING:
153
+
154
+By default, symbolic links are preserved as UnixFS symlink nodes that store
155
+the target path. Use --dereference-symlinks to resolve symlinks to their
156
+target content instead:
157
+
158
+ > ipfs add -r --dereference-symlinks ./mydir
159
+
160
+This resolves all symlinks, including CLI arguments and those found inside
161
+directories. Symlinks to files become regular file content, symlinks to
162
+directories are traversed and their contents are added.
163
+
164
CHUNKING EXAMPLES:
165
166
The chunker option, '-s', specifies the chunking strategy that dictates
@@ -200,11 +214,13 @@ https://github.com/ipfs/kubo/blob/master/docs/config.md#import
214
Options: []cmds.Option{
215
// Input Processing
216
cmds.OptionRecursivePath, // a builtin option that allows recursive paths (-r, --recursive)
203
- cmds.OptionDerefArgs, // a builtin option that resolves passed in filesystem links (--dereference-args)
217
+ cmds.OptionDerefArgs, // DEPRECATED: use --dereference-symlinks instead
218
cmds.OptionStdinName, // a builtin option that optionally allows wrapping stdin into a named file
219
cmds.OptionHidden,
220
cmds.OptionIgnore,
221
cmds.OptionIgnoreRules,
222
+ cmds.BoolOption(emptyDirsOptionName, "E", "Include empty directories in the import.").WithDefault(config.DefaultUnixFSIncludeEmptyDirs),
223
+ cmds.OptionDerefSymlinks, // resolve symlinks to their target content
224
// Output Control
225
cmds.BoolOption(quietOptionName, "q", "Write minimal output."),
226
cmds.BoolOption(quieterOptionName, "Q", "Write only final hash."),
@@ -274,7 +290,7 @@ https://github.com/ipfs/kubo/blob/master/docs/config.md#import
290
}
291
292
progress, _ := req.Options[progressOptionName].(bool)
277
- trickle, _ := req.Options[trickleOptionName].(bool)
293
+ trickle, trickleSet := req.Options[trickleOptionName].(bool)
294
wrap, _ := req.Options[wrapOptionName].(bool)
295
onlyHash, _ := req.Options[onlyHashOptionName].(bool)
296
silent, _ := req.Options[silentOptionName].(bool)
@@ -285,6 +301,7 @@ https://github.com/ipfs/kubo/blob/master/docs/config.md#import
301
maxFileLinks, maxFileLinksSet := req.Options[maxFileLinksOptionName].(int)
302
maxDirectoryLinks, maxDirectoryLinksSet := req.Options[maxDirectoryLinksOptionName].(int)
303
maxHAMTFanout, maxHAMTFanoutSet := req.Options[maxHAMTFanoutOptionName].(int)
304
+ var sizeEstimationMode uio.SizeEstimationMode
305
nocopy, _ := req.Options[noCopyOptionName].(bool)
306
fscache, _ := req.Options[fstoreCacheOptionName].(bool)
307
cidVer, cidVerSet := req.Options[cidVersionOptionName].(int)
@@ -312,6 +329,17 @@ https://github.com/ipfs/kubo/blob/master/docs/config.md#import
329
mtimeNsecs, _ := req.Options[mtimeNsecsOptionName].(uint)
330
fastProvideRoot, fastProvideRootSet := req.Options[fastProvideRootOptionName].(bool)
331
fastProvideWait, fastProvideWaitSet := req.Options[fastProvideWaitOptionName].(bool)
332
+ emptyDirs, _ := req.Options[emptyDirsOptionName].(bool)
333
+
334
+ // Note: --dereference-args is deprecated but still works for backwards compatibility.
335
+ // The help text marks it as DEPRECATED. Users should use --dereference-symlinks instead,
336
+ // which is a superset (resolves both CLI arg symlinks AND nested symlinks in directories).
337
+
338
+ // Wire --trickle from config
339
+ if !trickleSet && !cfg.Import.UnixFSDAGLayout.IsDefault() {
340
+ layout := cfg.Import.UnixFSDAGLayout.WithDefault(config.DefaultUnixFSDAGLayout)
341
+ trickle = layout == config.DAGLayoutTrickle
342
+ }
343
344
if chunker == "" {
345
chunker = cfg.Import.UnixFSChunker.WithDefault(config.DefaultUnixFSChunker)
@@ -348,6 +376,9 @@ https://github.com/ipfs/kubo/blob/master/docs/config.md#import
376
maxHAMTFanout = int(cfg.Import.UnixFSHAMTDirectoryMaxFanout.WithDefault(config.DefaultUnixFSHAMTDirectoryMaxFanout))
377
}
378
379
+ // SizeEstimationMode is always set from config (no CLI flag)
380
+ sizeEstimationMode = cfg.Import.HAMTSizeEstimationMode()
381
+
382
fastProvideRoot = config.ResolveBoolFromConfig(fastProvideRoot, fastProvideRootSet, cfg.Import.FastProvideRoot, config.DefaultFastProvideRoot)
383
fastProvideWait = config.ResolveBoolFromConfig(fastProvideWait, fastProvideWaitSet, cfg.Import.FastProvideWait, config.DefaultFastProvideWait)
384
@@ -409,6 +440,8 @@ https://github.com/ipfs/kubo/blob/master/docs/config.md#import
440
441
options.Unixfs.PreserveMode(preserveMode),
442
options.Unixfs.PreserveMtime(preserveMtime),
443
+
444
+ options.Unixfs.IncludeEmptyDirs(emptyDirs),
445
}
446
447
if mode != 0 {
@@ -441,6 +474,9 @@ https://github.com/ipfs/kubo/blob/master/docs/config.md#import
474
opts = append(opts, options.Unixfs.MaxHAMTFanout(maxHAMTFanout))
475
}
476
477
+ // SizeEstimationMode is always set from config
478
+ opts = append(opts, options.Unixfs.SizeEstimationMode(sizeEstimationMode))
479
+
480
if trickle {
481
opts = append(opts, options.Unixfs.Layout(options.TrickleLayout))
482
}
core/commands/files.go
+63
-15
@@ -28,6 +28,7 @@ import (
28
offline "github.com/ipfs/boxo/exchange/offline"
29
dag "github.com/ipfs/boxo/ipld/merkledag"
30
ft "github.com/ipfs/boxo/ipld/unixfs"
31
+ uio "github.com/ipfs/boxo/ipld/unixfs/io"
32
mfs "github.com/ipfs/boxo/mfs"
33
"github.com/ipfs/boxo/path"
34
cid "github.com/ipfs/go-cid"
@@ -499,7 +500,12 @@ being GC'ed.
500
return err
501
}
502
502
- prefix, err := getPrefixNew(req)
503
+ cfg, err := nd.Repo.Config()
504
+ if err != nil {
505
+ return err
506
+ }
507
+
508
+ prefix, err := getPrefixNew(req, &cfg.Import)
509
if err != nil {
510
return err
511
}
@@ -550,7 +556,9 @@ being GC'ed.
556
557
mkParents, _ := req.Options[filesParentsOptionName].(bool)
558
if mkParents {
553
- err := ensureContainingDirectoryExists(nd.FilesRoot, dst, prefix)
559
+ maxDirLinks := int(cfg.Import.UnixFSDirectoryMaxLinks.WithDefault(config.DefaultUnixFSDirectoryMaxLinks))
560
+ sizeEstimationMode := cfg.Import.HAMTSizeEstimationMode()
561
+ err := ensureContainingDirectoryExists(nd.FilesRoot, dst, prefix, maxDirLinks, &sizeEstimationMode)
562
if err != nil {
563
return err
564
}
@@ -989,9 +997,13 @@ stat' on the file or any of its ancestors.
997
WARNING:
998
999
The CID produced by 'files write' will be different from 'ipfs add' because
992
-'ipfs file write' creates a trickle-dag optimized for append-only operations
1000
+'ipfs files write' creates a trickle-dag optimized for append-only operations.
1001
See '--trickle' in 'ipfs add --help' for more information.
1002
1003
+NOTE: The 'Import.UnixFSFileMaxLinks' config option does not apply to this command.
1004
+Trickle DAG has a fixed internal structure optimized for append operations.
1005
+To use configurable max-links, use 'ipfs add' with balanced DAG layout.
1006
+
1007
If you want to add a file without modifying an existing one,
1008
use 'ipfs add' with '--to-files':
1009
@@ -1048,7 +1060,7 @@ See '--to-files' in 'ipfs add --help' for more information.
1060
rawLeaves = cfg.Import.UnixFSRawLeaves.WithDefault(config.DefaultUnixFSRawLeaves)
1061
}
1062
1051
- prefix, err := getPrefixNew(req)
1063
+ prefix, err := getPrefixNew(req, &cfg.Import)
1064
if err != nil {
1065
return err
1066
}
@@ -1059,7 +1071,9 @@ See '--to-files' in 'ipfs add --help' for more information.
1071
}
1072
1073
if mkParents {
1062
- err := ensureContainingDirectoryExists(nd.FilesRoot, path, prefix)
1074
+ maxDirLinks := int(cfg.Import.UnixFSDirectoryMaxLinks.WithDefault(config.DefaultUnixFSDirectoryMaxLinks))
1075
+ sizeEstimationMode := cfg.Import.HAMTSizeEstimationMode()
1076
+ err := ensureContainingDirectoryExists(nd.FilesRoot, path, prefix, maxDirLinks, &sizeEstimationMode)
1077
if err != nil {
1078
return err
1079
}
@@ -1163,6 +1177,11 @@ Examples:
1177
return err
1178
}
1179
1180
+ cfg, err := n.Repo.Config()
1181
+ if err != nil {
1182
+ return err
1183
+ }
1184
+
1185
dashp, _ := req.Options[filesParentsOptionName].(bool)
1186
dirtomake, err := checkPath(req.Arguments[0])
1187
if err != nil {
@@ -1175,16 +1194,21 @@ Examples:
1194
return err
1195
}
1196
1178
- prefix, err := getPrefix(req)
1197
+ prefix, err := getPrefix(req, &cfg.Import)
1198
if err != nil {
1199
return err
1200
}
1201
root := n.FilesRoot
1202
1203
+ maxDirLinks := int(cfg.Import.UnixFSDirectoryMaxLinks.WithDefault(config.DefaultUnixFSDirectoryMaxLinks))
1204
+ sizeEstimationMode := cfg.Import.HAMTSizeEstimationMode()
1205
+
1206
err = mfs.Mkdir(root, dirtomake, mfs.MkdirOpts{
1185
- Mkparents: dashp,
1186
- Flush: flush,
1187
- CidBuilder: prefix,
1207
+ Mkparents: dashp,
1208
+ Flush: flush,
1209
+ CidBuilder: prefix,
1210
+ MaxLinks: maxDirLinks,
1211
+ SizeEstimationMode: &sizeEstimationMode,
1212
})
1213
1214
return err
@@ -1262,7 +1286,9 @@ Change the CID version or hash function of the root node of a given path.
1286
1287
flush, _ := req.Options[filesFlushOptionName].(bool)
1288
1265
- prefix, err := getPrefix(req)
1289
+ // Note: files chcid is for explicitly changing CID format, so we don't
1290
+ // fall back to Import config here. If no options are provided, it does nothing.
1291
+ prefix, err := getPrefix(req, nil)
1292
if err != nil {
1293
return err
1294
}
@@ -1420,10 +1446,20 @@ func removePath(filesRoot *mfs.Root, path string, force bool, dashr bool) error
1446
return pdir.Flush()
1447
}
1448
1423
-func getPrefixNew(req *cmds.Request) (cid.Builder, error) {
1449
+func getPrefixNew(req *cmds.Request, importCfg *config.Import) (cid.Builder, error) {
1450
cidVer, cidVerSet := req.Options[filesCidVersionOptionName].(int)
1451
hashFunStr, hashFunSet := req.Options[filesHashOptionName].(string)
1452
1453
+ // Fall back to Import config if CLI options not set
1454
+ if !cidVerSet && importCfg != nil && !importCfg.CidVersion.IsDefault() {
1455
+ cidVer = int(importCfg.CidVersion.WithDefault(config.DefaultCidVersion))
1456
+ cidVerSet = true
1457
+ }
1458
+ if !hashFunSet && importCfg != nil && !importCfg.HashFunction.IsDefault() {
1459
+ hashFunStr = importCfg.HashFunction.WithDefault(config.DefaultHashFunction)
1460
+ hashFunSet = true
1461
+ }
1462
+
1463
if !cidVerSet && !hashFunSet {
1464
return nil, nil
1465
}
@@ -1449,10 +1485,20 @@ func getPrefixNew(req *cmds.Request) (cid.Builder, error) {
1485
return &prefix, nil
1486
}
1487
1452
-func getPrefix(req *cmds.Request) (cid.Builder, error) {
1488
+func getPrefix(req *cmds.Request, importCfg *config.Import) (cid.Builder, error) {
1489
cidVer, cidVerSet := req.Options[filesCidVersionOptionName].(int)
1490
hashFunStr, hashFunSet := req.Options[filesHashOptionName].(string)
1491
1492
+ // Fall back to Import config if CLI options not set
1493
+ if !cidVerSet && importCfg != nil && !importCfg.CidVersion.IsDefault() {
1494
+ cidVer = int(importCfg.CidVersion.WithDefault(config.DefaultCidVersion))
1495
+ cidVerSet = true
1496
+ }
1497
+ if !hashFunSet && importCfg != nil && !importCfg.HashFunction.IsDefault() {
1498
+ hashFunStr = importCfg.HashFunction.WithDefault(config.DefaultHashFunction)
1499
+ hashFunSet = true
1500
+ }
1501
+
1502
if !cidVerSet && !hashFunSet {
1503
return nil, nil
1504
}
@@ -1478,7 +1524,7 @@ func getPrefix(req *cmds.Request) (cid.Builder, error) {
1524
return &prefix, nil
1525
}
1526
1481
-func ensureContainingDirectoryExists(r *mfs.Root, path string, builder cid.Builder) error {
1527
+func ensureContainingDirectoryExists(r *mfs.Root, path string, builder cid.Builder, maxLinks int, sizeEstimationMode *uio.SizeEstimationMode) error {
1528
dirtomake := gopath.Dir(path)
1529
1530
if dirtomake == "/" {
@@ -1486,8 +1532,10 @@ func ensureContainingDirectoryExists(r *mfs.Root, path string, builder cid.Build
1532
}
1533
1534
return mfs.Mkdir(r, dirtomake, mfs.MkdirOpts{
1489
- Mkparents: true,
1490
- CidBuilder: builder,
1535
+ Mkparents: true,
1536
+ CidBuilder: builder,
1537
+ MaxLinks: maxLinks,
1538
+ SizeEstimationMode: sizeEstimationMode,
1539
})
1540
}
1541
core/coreapi/unixfs.go
+6
@@ -177,12 +177,18 @@ func (api *UnixfsAPI) Add(ctx context.Context, files files.Node, opts ...options
177
if settings.MaxHAMTFanoutSet {
178
fileAdder.MaxHAMTFanout = settings.MaxHAMTFanout
179
}
180
+ if settings.SizeEstimationModeSet {
181
+ fileAdder.SizeEstimationMode = settings.SizeEstimationMode
182
+ }
183
fileAdder.NoCopy = settings.NoCopy
184
fileAdder.CidBuilder = prefix
185
fileAdder.PreserveMode = settings.PreserveMode
186
fileAdder.PreserveMtime = settings.PreserveMtime
187
fileAdder.FileMode = settings.Mode
188
fileAdder.FileMtime = settings.Mtime
189
+ if settings.IncludeEmptyDirsSet {
190
+ fileAdder.IncludeEmptyDirs = settings.IncludeEmptyDirs
191
+ }
192
193
switch settings.Layout {
194
case options.BalancedLayout:
core/coreiface/options/unixfs.go
+42
-18
@@ -24,16 +24,18 @@ type UnixfsAddSettings struct {
24
CidVersion int
25
MhType uint64
26
27
- Inline bool
28
- InlineLimit int
29
- RawLeaves bool
30
- RawLeavesSet bool
31
- MaxFileLinks int
32
- MaxFileLinksSet bool
33
- MaxDirectoryLinks int
34
- MaxDirectoryLinksSet bool
35
- MaxHAMTFanout int
36
- MaxHAMTFanoutSet bool
27
+ Inline bool
28
+ InlineLimit int
29
+ RawLeaves bool
30
+ RawLeavesSet bool
31
+ MaxFileLinks int
32
+ MaxFileLinksSet bool
33
+ MaxDirectoryLinks int
34
+ MaxDirectoryLinksSet bool
35
+ MaxHAMTFanout int
36
+ MaxHAMTFanoutSet bool
37
+ SizeEstimationMode *io.SizeEstimationMode
38
+ SizeEstimationModeSet bool
39
40
Chunker string
41
Layout Layout
@@ -48,10 +50,12 @@ type UnixfsAddSettings struct {
50
Silent bool
51
Progress bool
52
51
- PreserveMode bool
52
- PreserveMtime bool
53
- Mode os.FileMode
54
- Mtime time.Time
53
+ PreserveMode bool
54
+ PreserveMtime bool
55
+ Mode os.FileMode
56
+ Mtime time.Time
57
+ IncludeEmptyDirs bool
58
+ IncludeEmptyDirsSet bool
59
}
60
61
type UnixfsLsSettings struct {
@@ -93,10 +97,12 @@ func UnixfsAddOptions(opts ...UnixfsAddOption) (*UnixfsAddSettings, cid.Prefix,
97
Silent: false,
98
Progress: false,
99
96
- PreserveMode: false,
97
- PreserveMtime: false,
98
- Mode: 0,
99
- Mtime: time.Time{},
100
+ PreserveMode: false,
101
+ PreserveMtime: false,
102
+ Mode: 0,
103
+ Mtime: time.Time{},
104
+ IncludeEmptyDirs: true, // default: include empty directories
105
+ IncludeEmptyDirsSet: false,
106
}
107
108
for _, opt := range opts {
@@ -235,6 +241,15 @@ func (unixfsOpts) MaxHAMTFanout(n int) UnixfsAddOption {
241
}
242
}
243
244
+// SizeEstimationMode specifies how directory size is estimated for HAMT sharding decisions.
245
+func (unixfsOpts) SizeEstimationMode(mode io.SizeEstimationMode) UnixfsAddOption {
246
+ return func(settings *UnixfsAddSettings) error {
247
+ settings.SizeEstimationMode = &mode
248
+ settings.SizeEstimationModeSet = true
249
+ return nil
250
+ }
251
+}
252
+
253
// Inline tells the adder to inline small blocks into CIDs
254
func (unixfsOpts) Inline(enable bool) UnixfsAddOption {
255
return func(settings *UnixfsAddSettings) error {
@@ -396,3 +411,12 @@ func (unixfsOpts) Mtime(seconds int64, nsecs uint32) UnixfsAddOption {
411
return nil
412
}
413
}
414
+
415
+// IncludeEmptyDirs tells the adder to include empty directories in the DAG
416
+func (unixfsOpts) IncludeEmptyDirs(include bool) UnixfsAddOption {
417
+ return func(settings *UnixfsAddSettings) error {
418
+ settings.IncludeEmptyDirs = include
419
+ settings.IncludeEmptyDirsSet = true
420
+ return nil
421
+ }
422
+}
core/coreunix/add.go
+89
-62
@@ -26,6 +26,7 @@ import (
26
"github.com/ipfs/go-cid"
27
ipld "github.com/ipfs/go-ipld-format"
28
logging "github.com/ipfs/go-log/v2"
29
+ "github.com/ipfs/kubo/config"
30
coreiface "github.com/ipfs/kubo/core/coreiface"
31
32
"github.com/ipfs/kubo/tracing"
@@ -52,49 +53,52 @@ func NewAdder(ctx context.Context, p pin.Pinner, bs bstore.GCLocker, ds ipld.DAG
53
bufferedDS := ipld.NewBufferedDAG(ctx, ds)
54
55
return &Adder{
55
- ctx: ctx,
56
- pinning: p,
57
- gcLocker: bs,
58
- dagService: ds,
59
- bufferedDS: bufferedDS,
60
- Progress: false,
61
- Pin: true,
62
- Trickle: false,
63
- MaxLinks: ihelper.DefaultLinksPerBlock,
64
- MaxHAMTFanout: uio.DefaultShardWidth,
65
- Chunker: "",
56
+ ctx: ctx,
57
+ pinning: p,
58
+ gcLocker: bs,
59
+ dagService: ds,
60
+ bufferedDS: bufferedDS,
61
+ Progress: false,
62
+ Pin: true,
63
+ Trickle: false,
64
+ MaxLinks: ihelper.DefaultLinksPerBlock,
65
+ MaxHAMTFanout: uio.DefaultShardWidth,
66
+ Chunker: "",
67
+ IncludeEmptyDirs: config.DefaultUnixFSIncludeEmptyDirs,
68
}, nil
69
}
70
71
// Adder holds the switches passed to the `add` command.
72
type Adder struct {
71
- ctx context.Context
72
- pinning pin.Pinner
73
- gcLocker bstore.GCLocker
74
- dagService ipld.DAGService
75
- bufferedDS *ipld.BufferedDAG
76
- Out chan<- interface{}
77
- Progress bool
78
- Pin bool
79
- PinName string
80
- Trickle bool
81
- RawLeaves bool
82
- MaxLinks int
83
- MaxDirectoryLinks int
84
- MaxHAMTFanout int
85
- Silent bool
86
- NoCopy bool
87
- Chunker string
88
- mroot *mfs.Root
89
- unlocker bstore.Unlocker
90
- tempRoot cid.Cid
91
- CidBuilder cid.Builder
92
- liveNodes uint64
93
-
94
- PreserveMode bool
95
- PreserveMtime bool
96
- FileMode os.FileMode
97
- FileMtime time.Time
73
+ ctx context.Context
74
+ pinning pin.Pinner
75
+ gcLocker bstore.GCLocker
76
+ dagService ipld.DAGService
77
+ bufferedDS *ipld.BufferedDAG
78
+ Out chan<- interface{}
79
+ Progress bool
80
+ Pin bool
81
+ PinName string
82
+ Trickle bool
83
+ RawLeaves bool
84
+ MaxLinks int
85
+ MaxDirectoryLinks int
86
+ MaxHAMTFanout int
87
+ SizeEstimationMode *uio.SizeEstimationMode
88
+ Silent bool
89
+ NoCopy bool
90
+ Chunker string
91
+ mroot *mfs.Root
92
+ unlocker bstore.Unlocker
93
+ tempRoot cid.Cid
94
+ CidBuilder cid.Builder
95
+ liveNodes uint64
96
+
97
+ PreserveMode bool
98
+ PreserveMtime bool
99
+ FileMode os.FileMode
100
+ FileMtime time.Time
101
+ IncludeEmptyDirs bool
102
}
103
104
func (adder *Adder) mfsRoot() (*mfs.Root, error) {
@@ -104,9 +108,10 @@ func (adder *Adder) mfsRoot() (*mfs.Root, error) {
108
109
// Note, this adds it to DAGService already.
110
mr, err := mfs.NewEmptyRoot(adder.ctx, adder.dagService, nil, nil, mfs.MkdirOpts{
107
- CidBuilder: adder.CidBuilder,
108
- MaxLinks: adder.MaxDirectoryLinks,
109
- MaxHAMTFanout: adder.MaxHAMTFanout,
111
+ CidBuilder: adder.CidBuilder,
112
+ MaxLinks: adder.MaxDirectoryLinks,
113
+ MaxHAMTFanout: adder.MaxHAMTFanout,
114
+ SizeEstimationMode: adder.SizeEstimationMode,
115
})
116
if err != nil {
117
return nil, err
@@ -270,11 +275,12 @@ func (adder *Adder) addNode(node ipld.Node, path string) error {
275
dir := gopath.Dir(path)
276
if dir != "." {
277
opts := mfs.MkdirOpts{
273
- Mkparents: true,
274
- Flush: false,
275
- CidBuilder: adder.CidBuilder,
276
- MaxLinks: adder.MaxDirectoryLinks,
277
- MaxHAMTFanout: adder.MaxHAMTFanout,
278
+ Mkparents: true,
279
+ Flush: false,
280
+ CidBuilder: adder.CidBuilder,
281
+ MaxLinks: adder.MaxDirectoryLinks,
282
+ MaxHAMTFanout: adder.MaxHAMTFanout,
283
+ SizeEstimationMode: adder.SizeEstimationMode,
284
}
285
if err := mfs.Mkdir(mr, dir, opts); err != nil {
286
return err
@@ -480,15 +486,34 @@ func (adder *Adder) addFile(path string, file files.File) error {
486
func (adder *Adder) addDir(ctx context.Context, path string, dir files.Directory, toplevel bool) error {
487
log.Infof("adding directory: %s", path)
488
489
+ // Peek at first entry to check if directory is empty.
490
+ // We advance the iterator once here and continue from this position
491
+ // in the processing loop below. This avoids allocating a slice to
492
+ // collect all entries just to check for emptiness.
493
+ it := dir.Entries()
494
+ hasEntry := it.Next()
495
+ if !hasEntry {
496
+ if err := it.Err(); err != nil {
497
+ return err
498
+ }
499
+ // Directory is empty. Skip it unless IncludeEmptyDirs is set or
500
+ // this is the toplevel directory (we always include the root).
501
+ if !adder.IncludeEmptyDirs && !toplevel {
502
+ log.Debugf("skipping empty directory: %s", path)
503
+ return nil
504
+ }
505
+ }
506
+
507
// if we need to store mode or modification time then create a new root which includes that data
508
if toplevel && (adder.FileMode != 0 || !adder.FileMtime.IsZero()) {
509
mr, err := mfs.NewEmptyRoot(ctx, adder.dagService, nil, nil,
510
mfs.MkdirOpts{
487
- CidBuilder: adder.CidBuilder,
488
- MaxLinks: adder.MaxDirectoryLinks,
489
- MaxHAMTFanout: adder.MaxHAMTFanout,
490
- ModTime: adder.FileMtime,
491
- Mode: adder.FileMode,
511
+ CidBuilder: adder.CidBuilder,
512
+ MaxLinks: adder.MaxDirectoryLinks,
513
+ MaxHAMTFanout: adder.MaxHAMTFanout,
514
+ ModTime: adder.FileMtime,
515
+ Mode: adder.FileMode,
516
+ SizeEstimationMode: adder.SizeEstimationMode,
517
})
518
if err != nil {
519
return err
@@ -502,26 +527,28 @@ func (adder *Adder) addDir(ctx context.Context, path string, dir files.Directory
527
return err
528
}
529
err = mfs.Mkdir(mr, path, mfs.MkdirOpts{
505
- Mkparents: true,
506
- Flush: false,
507
- CidBuilder: adder.CidBuilder,
508
- Mode: adder.FileMode,
509
- ModTime: adder.FileMtime,
510
- MaxLinks: adder.MaxDirectoryLinks,
511
- MaxHAMTFanout: adder.MaxHAMTFanout,
530
+ Mkparents: true,
531
+ Flush: false,
532
+ CidBuilder: adder.CidBuilder,
533
+ Mode: adder.FileMode,
534
+ ModTime: adder.FileMtime,
535
+ MaxLinks: adder.MaxDirectoryLinks,
536
+ MaxHAMTFanout: adder.MaxHAMTFanout,
537
+ SizeEstimationMode: adder.SizeEstimationMode,
538
})
539
if err != nil {
540
return err
541
}
542
}
543
518
- it := dir.Entries()
519
- for it.Next() {
544
+ // Process directory entries. The iterator was already advanced once above
545
+ // to peek for emptiness, so we start from that position.
546
+ for hasEntry {
547
fpath := gopath.Join(path, it.Name())
521
- err := adder.addFileNode(ctx, fpath, it.Node(), false)
522
- if err != nil {
548
+ if err := adder.addFileNode(ctx, fpath, it.Node(), false); err != nil {
549
return err
550
}
551
+ hasEntry = it.Next()
552
}
553
554
return it.Err()
core/node/core.go
+18
-1
@@ -243,7 +243,24 @@ func Files(strategy string) func(mctx helpers.MetricsCtx, lc fx.Lifecycle, repo
243
prov = nil
244
}
245
246
- root, err := mfs.NewRoot(ctx, dag, nd, pf, prov)
246
+ // Get configured settings from Import config
247
+ cfg, err := repo.Config()
248
+ if err != nil {
249
+ return nil, fmt.Errorf("failed to get config: %w", err)
250
+ }
251
+ chunkerGen := cfg.Import.UnixFSSplitterFunc()
252
+ maxDirLinks := int(cfg.Import.UnixFSDirectoryMaxLinks.WithDefault(config.DefaultUnixFSDirectoryMaxLinks))
253
+ maxHAMTFanout := int(cfg.Import.UnixFSHAMTDirectoryMaxFanout.WithDefault(config.DefaultUnixFSHAMTDirectoryMaxFanout))
254
+ hamtShardingSize := int(cfg.Import.UnixFSHAMTDirectorySizeThreshold.WithDefault(config.DefaultUnixFSHAMTDirectorySizeThreshold))
255
+ sizeEstimationMode := cfg.Import.HAMTSizeEstimationMode()
256
+
257
+ root, err := mfs.NewRoot(ctx, dag, nd, pf, prov,
258
+ mfs.WithChunker(chunkerGen),
259
+ mfs.WithMaxLinks(maxDirLinks),
260
+ mfs.WithMaxHAMTFanout(maxHAMTFanout),
261
+ mfs.WithHAMTShardingSize(hamtShardingSize),
262
+ mfs.WithSizeEstimationMode(sizeEstimationMode),
263
+ )
264
if err != nil {
265
return nil, fmt.Errorf("failed to initialize MFS root from %s stored at %s: %w. "+
266
"If corrupted, use 'ipfs files chroot' to reset (see --help)", nd.Cid(), FilesRootDatastoreKey, err)
core/node/groups.go
+5
-4
@@ -438,12 +438,13 @@ func IPFS(ctx context.Context, bcfg *BuildCfg) fx.Option {
438
return fx.Error(err)
439
}
440
441
- // Auto-sharding settings
442
- shardSingThresholdInt := cfg.Import.UnixFSHAMTDirectorySizeThreshold.WithDefault(config.DefaultUnixFSHAMTDirectorySizeThreshold)
441
+ // Directory sharding settings from Import config.
442
+ // These globals affect both `ipfs add` and MFS (`ipfs files` API).
443
+ shardSizeThreshold := cfg.Import.UnixFSHAMTDirectorySizeThreshold.WithDefault(config.DefaultUnixFSHAMTDirectorySizeThreshold)
444
shardMaxFanout := cfg.Import.UnixFSHAMTDirectoryMaxFanout.WithDefault(config.DefaultUnixFSHAMTDirectoryMaxFanout)
444
- // TODO: avoid overriding this globally, see if we can extend Directory interface like Get/SetMaxLinks from https://github.com/ipfs/boxo/pull/906
445
- uio.HAMTShardingSize = int(shardSingThresholdInt)
445
+ uio.HAMTShardingSize = int(shardSizeThreshold)
446
uio.DefaultShardWidth = int(shardMaxFanout)
447
+ uio.HAMTSizeEstimation = cfg.Import.HAMTSizeEstimationMode()
448
449
providerStrategy := cfg.Provide.Strategy.WithDefault(config.DefaultProvideStrategy)
450
docs/changelogs/v0.40.md
+34
@@ -10,6 +10,7 @@ This release was brought to you by the [Shipyard](https://ipshipyard.com/) team.
10
11
- [Overview](#overview)
12
- [🔦 Highlights](#-highlights)
13
+ - [🔢 UnixFS CID Profiles (IPIP-499)](#-unixfs-cid-profiles-ipip-499)
14
- [🧹 Automatic cleanup of interrupted imports](#-automatic-cleanup-of-interrupted-imports)
15
- [Routing V1 HTTP API now exposed by default](#routing-v1-http-api-now-exposed-by-default)
16
- [Track total size when adding pins](#track-total-size-when-adding-pins)
@@ -34,6 +35,39 @@ This release was brought to you by the [Shipyard](https://ipshipyard.com/) team.
35
36
### 🔦 Highlights
37
38
+#### 🔢 UnixFS CID Profiles (IPIP-499)
39
+
40
+[IPIP-499](https://github.com/ipfs/specs/pull/499) CID Profiles are presets that pin down how files get split into blocks and organized into directories. Useful when you need the same CID for the same data across different software or versions.
41
+
42
+**New configuration [profiles](https://github.com/ipfs/kubo/blob/master/docs/config.md#profiles)**
43
+
44
+- `unixfs-v1-2025`: modern CIDv1 profile with improved defaults
45
+- `unixfs-v0-2015` (alias `legacy-cid-v0`): best-effort legacy CIDv0 behavior
46
+
47
+Apply with: `ipfs config profile apply unixfs-v1-2025`
48
+
49
+The `test-cid-v1` and `test-cid-v1-wide` profiles have been removed. Use `unixfs-v1-2025` or manually set specific `Import.*` settings instead.
50
+
51
+**New [`Import.*`](https://github.com/ipfs/kubo/blob/master/docs/config.md#import) options**
52
+
53
+- `Import.UnixFSHAMTDirectorySizeEstimation`: estimation mode (`links`, `block`, or `disabled`)
54
+- `Import.UnixFSDAGLayout`: DAG layout (`balanced` or `trickle`)
55
+
56
+**New [`ipfs add`](https://docs.ipfs.tech/reference/kubo/cli/#ipfs-add) CLI flags**
57
+
58
+- `--dereference-symlinks` resolves all symlinks to their target content, replacing the deprecated `--dereference-args` which only resolved CLI argument symlinks
59
+- `--empty-dirs` / `-E` controls inclusion of empty directories (default: true)
60
+- `--hidden` / `-H` includes hidden files (default: false)
61
+- `--trickle` implicit default can be adjusted via `Import.UnixFSDAGLayout`
62
+
63
+**`ipfs files write` fix for CIDv1 directories**
64
+
65
+When writing to MFS directories that use CIDv1 (via `--cid-version=1` or `ipfs files chcid`), single-block files now produce raw block CIDs (like `bafkrei...`), matching the behavior of `ipfs add --raw-leaves`. Previously, MFS would wrap single-block files in dag-pb even when raw leaves were enabled. CIDv0 directories continue to use dag-pb.
66
+
67
+**HAMT Threshold Fix**
68
+
69
+HAMT directory sharding threshold changed from `>=` to `>` to match the Go docs and JS implementation ([ipfs/boxo@6707376](https://github.com/ipfs/boxo/commit/6707376002a3d4ba64895749ce9be2e00d265ed5)). A directory exactly at 256 KiB now stays as a basic directory instead of converting to HAMT. This is a theoretical breaking change, but unlikely to impact real-world users as it requires a directory to be exactly at the threshold boundary. If you depend on the old behavior, adjust [`Import.UnixFSHAMTShardingSize`](https://github.com/ipfs/kubo/blob/master/docs/config.md#importunixfshamtshardingsize) to be 1 byte lower.
70
+
71
#### 🧹 Automatic cleanup of interrupted imports
72
73
If you cancel `ipfs add` or `ipfs dag import` mid-operation, Kubo now automatically cleans up incomplete data on the next daemon start. Previously, interrupted imports would leave orphan blocks in your repository that were difficult to identify and remove without pins and running explicit garbage collection.
docs/config.md
+60
-27
@@ -242,6 +242,8 @@ config file at runtime.
242
- [`Import.UnixFSDirectoryMaxLinks`](#importunixfsdirectorymaxlinks)
243
- [`Import.UnixFSHAMTDirectoryMaxFanout`](#importunixfshamtdirectorymaxfanout)
244
- [`Import.UnixFSHAMTDirectorySizeThreshold`](#importunixfshamtdirectorysizethreshold)
245
+ - [`Import.UnixFSHAMTDirectorySizeEstimation`](#importunixfshamtdirectorysizeestimation)
246
+ - [`Import.UnixFSDAGLayout`](#importunixfsdaglayout)
247
- [`Version`](#version)
248
- [`Version.AgentSuffix`](#versionagentsuffix)
249
- [`Version.SwarmCheckEnabled`](#versionswarmcheckenabled)
@@ -263,9 +265,9 @@ config file at runtime.
265
- [`lowpower` profile](#lowpower-profile)
266
- [`announce-off` profile](#announce-off-profile)
267
- [`announce-on` profile](#announce-on-profile)
268
+ - [`unixfs-v0-2015` profile](#unixfs-v0-2015-profile)
269
- [`legacy-cid-v0` profile](#legacy-cid-v0-profile)
267
- - [`test-cid-v1` profile](#test-cid-v1-profile)
268
- - [`test-cid-v1-wide` profile](#test-cid-v1-wide-profile)
270
+ - [`unixfs-v1-2025` profile](#unixfs-v1-2025-profile)
271
- [Security](#security)
272
- [Port and Network Exposure](#port-and-network-exposure)
273
- [Security Best Practices](#security-best-practices)
@@ -3656,9 +3658,11 @@ Type: `flag`
3658
3659
## `Import`
3660
3659
-Options to configure the default options used for ingesting data, in commands such as `ipfs add` or `ipfs block put`. All affected commands are detailed per option.
3661
+Options to configure the default parameters used for ingesting data, in commands such as `ipfs add` or `ipfs block put`. All affected commands are detailed per option.
3662
3661
-Note that using flags will override the options defined here.
3663
+These options implement [IPIP-499: UnixFS CID Profiles](https://github.com/ipfs/specs/pull/499) for reproducible CID generation across IPFS implementations. Instead of configuring individual options, you can apply a predefined profile with `ipfs config profile apply <profile-name>`. See [Profiles](#profiles) for available options like `unixfs-v1-2025`.
3664
+
3665
+Note that using CLI flags will override the options defined here.
3666
3667
### `Import.CidVersion`
3668
@@ -3838,6 +3842,42 @@ Default: `256KiB` (may change, inspect `DefaultUnixFSHAMTDirectorySizeThreshold`
3842
3843
Type: [`optionalBytes`](#optionalbytes)
3844
3845
+### `Import.UnixFSHAMTDirectorySizeEstimation`
3846
+
3847
+Controls how directory size is estimated when deciding whether to switch
3848
+from a basic UnixFS directory to HAMT sharding.
3849
+
3850
+Accepted values:
3851
+
3852
+- `links` (default): Legacy estimation using sum of link names and CID byte lengths.
3853
+- `block`: Full serialized dag-pb block size for accurate threshold decisions.
3854
+- `disabled`: Disable HAMT sharding entirely (directories always remain basic).
3855
+
3856
+The `block` estimation is recommended for new profiles as it provides more
3857
+accurate threshold decisions and better cross-implementation consistency.
3858
+See [IPIP-499](https://github.com/ipfs/specs/pull/499) for more details.
3859
+
3860
+Commands affected: `ipfs add`
3861
+
3862
+Default: `links`
3863
+
3864
+Type: `optionalString`
3865
+
3866
+### `Import.UnixFSDAGLayout`
3867
+
3868
+Controls the DAG layout used when chunking files.
3869
+
3870
+Accepted values:
3871
+
3872
+- `balanced` (default): Balanced DAG layout with uniform leaf depth.
3873
+- `trickle`: Trickle DAG layout optimized for streaming.
3874
+
3875
+Commands affected: `ipfs add`
3876
+
3877
+Default: `balanced`
3878
+
3879
+Type: `optionalString`
3880
+
3881
## `Version`
3882
3883
Options to configure agent version announced to the swarm, and leveraging
@@ -3881,7 +3921,7 @@ applied with the `--profile` flag to `ipfs init` or with the `ipfs config profil
3921
apply` command. When a profile is applied a backup of the configuration file
3922
will be created in `$IPFS_PATH`.
3923
3884
-Configuration profiles can be applied additively. For example, both the `test-cid-v1` and `lowpower` profiles can be applied one after the other.
3924
+Configuration profiles can be applied additively. For example, both the `unixfs-v1-2025` and `lowpower` profiles can be applied one after the other.
3925
The available configuration profiles are listed below. You can also find them
3926
documented in `ipfs config profile --help`.
3927
@@ -4038,42 +4078,35 @@ Disables [Provide](#provide) system (and announcing to Amino DHT).
4078
4079
(Re-)enables [Provide](#provide) system (reverts [`announce-off` profile](#announce-off-profile)).
4080
4041
-### `legacy-cid-v0` profile
4081
+### `unixfs-v0-2015` profile
4082
4043
-Makes UnixFS import (`ipfs add`) produce legacy CIDv0 with no raw leaves, sha2-256 and 256 KiB chunks.
4083
+Legacy UnixFS import profile for backward-compatible CID generation.
4084
+Produces CIDv0 with no raw leaves, sha2-256, 256 KiB chunks, and
4085
+link-based HAMT size estimation.
4086
4087
See <https://github.com/ipfs/kubo/blob/master/config/profile.go> for exact [`Import.*`](#import) settings.
4088
4089
> [!NOTE]
4048
-> This profile is provided for legacy users and should not be used for new projects.
4049
-
4050
-### `test-cid-v1` profile
4051
-
4052
-Makes UnixFS import (`ipfs add`) produce modern CIDv1 with raw leaves, sha2-256
4053
-and 1 MiB chunks (max 174 links per file, 256 per HAMT node, switch dir to HAMT
4054
-above 256KiB).
4090
+> Use only when legacy CIDs are required. For new projects, use [`unixfs-v1-2025`](#unixfs-v1-2025-profile).
4091
+>
4092
+> See [IPIP-499](https://github.com/ipfs/specs/pull/499) for more details.
4093
4056
-See <https://github.com/ipfs/kubo/blob/master/config/profile.go> for exact [`Import.*`](#import) settings.
4094
+### `legacy-cid-v0` profile
4095
4058
-> [!NOTE]
4059
-> [`Import.*`](#import) settings applied by this profile MAY change in future release. Provided for testing purposes.
4060
->
4061
-> Follow [kubo#4143](https://github.com/ipfs/kubo/issues/4143) for more details,
4062
-> and provide feedback in [discuss.ipfs.tech/t/should-we-profile-cids](https://discuss.ipfs.tech/t/should-we-profile-cids/18507) or [ipfs/specs#499](https://github.com/ipfs/specs/pull/499).
4096
+Alias for [`unixfs-v0-2015`](#unixfs-v0-2015-profile) profile.
4097
4064
-### `test-cid-v1-wide` profile
4098
+### `unixfs-v1-2025` profile
4099
4066
-Makes UnixFS import (`ipfs add`) produce modern CIDv1 with raw leaves, sha2-256
4067
-and 1 MiB chunks and wider file DAGs (max 1024 links per every node type,
4068
-switch dir to HAMT above 1MiB).
4100
+Recommended UnixFS import profile for cross-implementation CID determinism.
4101
+Uses CIDv1, raw leaves, sha2-256, 1 MiB chunks, 1024 links per file node,
4102
+256 HAMT fanout, and block-based size estimation for HAMT threshold.
4103
4104
See <https://github.com/ipfs/kubo/blob/master/config/profile.go> for exact [`Import.*`](#import) settings.
4105
4106
> [!NOTE]
4073
-> [`Import.*`](#import) settings applied by this profile MAY change in future release. Provided for testing purposes.
4107
+> This profile ensures CID consistency across different IPFS implementations.
4108
>
4075
-> Follow [kubo#4143](https://github.com/ipfs/kubo/issues/4143) for more details,
4076
-> and provide feedback in [discuss.ipfs.tech/t/should-we-profile-cids](https://discuss.ipfs.tech/t/should-we-profile-cids/18507) or [ipfs/specs#499](https://github.com/ipfs/specs/pull/499).
4109
+> See [IPIP-499](https://github.com/ipfs/specs/pull/499) for more details.
4110
4111
## Security
4112
docs/examples/kubo-as-a-library/go.mod
+2
-2
@@ -7,7 +7,7 @@ go 1.25
7
replace github.com/ipfs/kubo => ./../../..
8
9
require (
10
- github.com/ipfs/boxo v0.36.1-0.20260204011824-2688767ff981
10
+ github.com/ipfs/boxo v0.36.1-0.20260204203152-f188f79fd412
11
github.com/ipfs/kubo v0.0.0-00010101000000-000000000000
12
github.com/libp2p/go-libp2p v0.47.0
13
github.com/multiformats/go-multiaddr v0.16.1
@@ -85,7 +85,7 @@ require (
85
github.com/ipfs/go-ds-pebble v0.5.9 // indirect
86
github.com/ipfs/go-dsqueue v0.1.2 // indirect
87
github.com/ipfs/go-fs-lock v0.1.1 // indirect
88
- github.com/ipfs/go-ipfs-cmds v0.15.1-0.20260203151407-4b3827ebb483 // indirect
88
+ github.com/ipfs/go-ipfs-cmds v0.15.1-0.20260204204540-af9bcbaf5709 // indirect
89
github.com/ipfs/go-ipfs-ds-help v1.1.1 // indirect
90
github.com/ipfs/go-ipfs-pq v0.0.4 // indirect
91
github.com/ipfs/go-ipfs-redirects-file v0.1.2 // indirect
docs/examples/kubo-as-a-library/go.sum
+4
-4
@@ -267,8 +267,8 @@ github.com/ipfs-shipyard/nopfs/ipfs v0.25.0 h1:OqNqsGZPX8zh3eFMO8Lf8EHRRnSGBMqcd
267
github.com/ipfs-shipyard/nopfs/ipfs v0.25.0/go.mod h1:BxhUdtBgOXg1B+gAPEplkg/GpyTZY+kCMSfsJvvydqU=
268
github.com/ipfs/bbloom v0.0.4 h1:Gi+8EGJ2y5qiD5FbsbpX/TMNcJw8gSqr7eyjHa4Fhvs=
269
github.com/ipfs/bbloom v0.0.4/go.mod h1:cS9YprKXpoZ9lT0n/Mw/a6/aFV6DTjTLYHeA+gyqMG0=
270
-github.com/ipfs/boxo v0.36.1-0.20260204011824-2688767ff981 h1:Q3XjjicNTpok8gD0WwbLYZpmbRoykNTiCLbpj3EjnPc=
271
-github.com/ipfs/boxo v0.36.1-0.20260204011824-2688767ff981/go.mod h1:92hnRXfP5ScKEIqlq9Ns7LR1dFXEVADKWVGH0fjk83k=
270
+github.com/ipfs/boxo v0.36.1-0.20260204203152-f188f79fd412 h1:nfRIkMIhetCWD8jw5ya+FY+jn9ii2c+U5gdkmSS4L1Q=
271
+github.com/ipfs/boxo v0.36.1-0.20260204203152-f188f79fd412/go.mod h1:92hnRXfP5ScKEIqlq9Ns7LR1dFXEVADKWVGH0fjk83k=
272
github.com/ipfs/go-bitfield v1.1.0 h1:fh7FIo8bSwaJEh6DdTWbCeZ1eqOaOkKFI74SCnsWbGA=
273
github.com/ipfs/go-bitfield v1.1.0/go.mod h1:paqf1wjq/D2BBmzfTVFlJQ9IlFOZpg422HL0HqsGWHU=
274
github.com/ipfs/go-block-format v0.0.3/go.mod h1:4LmD4ZUw0mhO+JSKdpWwrzATiEfM7WWgQ8H5l6P8MVk=
@@ -303,8 +303,8 @@ github.com/ipfs/go-dsqueue v0.1.2 h1:jBMsgvT9Pj9l3cqI0m5jYpW/aWDYkW4Us6EuzrcSGbs
303
github.com/ipfs/go-dsqueue v0.1.2/go.mod h1:OU94YuMVUIF/ctR7Ysov9PI4gOa2XjPGN9nd8imSv78=
304
github.com/ipfs/go-fs-lock v0.1.1 h1:TecsP/Uc7WqYYatasreZQiP9EGRy4ZnKoG4yXxR33nw=
305
github.com/ipfs/go-fs-lock v0.1.1/go.mod h1:2goSXMCw7QfscHmSe09oXiR34DQeUdm+ei+dhonqly0=
306
-github.com/ipfs/go-ipfs-cmds v0.15.1-0.20260203151407-4b3827ebb483 h1:FnQqL92YxPX08/dcqE4cCSqEzwVGSdj2wprWHX+cUtM=
307
-github.com/ipfs/go-ipfs-cmds v0.15.1-0.20260203151407-4b3827ebb483/go.mod h1:YmhRbpaLKg40i9Ogj2+L41tJ+8x50fF8u1FJJD/WNhc=
306
+github.com/ipfs/go-ipfs-cmds v0.15.1-0.20260204204540-af9bcbaf5709 h1:0JiurWPnR7ZtjYW8XdfThOcOU5WlVVGQ1JY4FHHgyu8=
307
+github.com/ipfs/go-ipfs-cmds v0.15.1-0.20260204204540-af9bcbaf5709/go.mod h1:yZeTCte5zTH66bbEpLPkSog3/ImppCD00DMP7NjYmys=
308
github.com/ipfs/go-ipfs-delay v0.0.0-20181109222059-70721b86a9a8/go.mod h1:8SP1YXK1M1kXuc4KJZINY3TQQ03J2rwBG9QfXmbRPrw=
309
github.com/ipfs/go-ipfs-delay v0.0.1 h1:r/UXYyRcddO6thwOnhiznIAiSvxMECGgtv35Xs1IeRQ=
310
github.com/ipfs/go-ipfs-delay v0.0.1/go.mod h1:8SP1YXK1M1kXuc4KJZINY3TQQ03J2rwBG9QfXmbRPrw=
go.mod
+2
-2
@@ -21,7 +21,7 @@ require (
21
github.com/hashicorp/go-version v1.8.0
22
github.com/ipfs-shipyard/nopfs v0.0.14
23
github.com/ipfs-shipyard/nopfs/ipfs v0.25.0
24
- github.com/ipfs/boxo v0.36.1-0.20260204011824-2688767ff981
24
+ github.com/ipfs/boxo v0.36.1-0.20260204203152-f188f79fd412
25
github.com/ipfs/go-block-format v0.2.3
26
github.com/ipfs/go-cid v0.6.0
27
github.com/ipfs/go-cidutil v0.1.0
@@ -33,7 +33,7 @@ require (
33
github.com/ipfs/go-ds-measure v0.2.2
34
github.com/ipfs/go-ds-pebble v0.5.9
35
github.com/ipfs/go-fs-lock v0.1.1
36
- github.com/ipfs/go-ipfs-cmds v0.15.1-0.20260203151407-4b3827ebb483
36
+ github.com/ipfs/go-ipfs-cmds v0.15.1-0.20260204204540-af9bcbaf5709
37
github.com/ipfs/go-ipld-cbor v0.2.1
38
github.com/ipfs/go-ipld-format v0.6.3
39
github.com/ipfs/go-ipld-git v0.1.1
go.sum
+4
-4
@@ -337,8 +337,8 @@ github.com/ipfs-shipyard/nopfs/ipfs v0.25.0 h1:OqNqsGZPX8zh3eFMO8Lf8EHRRnSGBMqcd
337
github.com/ipfs-shipyard/nopfs/ipfs v0.25.0/go.mod h1:BxhUdtBgOXg1B+gAPEplkg/GpyTZY+kCMSfsJvvydqU=
338
github.com/ipfs/bbloom v0.0.4 h1:Gi+8EGJ2y5qiD5FbsbpX/TMNcJw8gSqr7eyjHa4Fhvs=
339
github.com/ipfs/bbloom v0.0.4/go.mod h1:cS9YprKXpoZ9lT0n/Mw/a6/aFV6DTjTLYHeA+gyqMG0=
340
-github.com/ipfs/boxo v0.36.1-0.20260204011824-2688767ff981 h1:Q3XjjicNTpok8gD0WwbLYZpmbRoykNTiCLbpj3EjnPc=
341
-github.com/ipfs/boxo v0.36.1-0.20260204011824-2688767ff981/go.mod h1:92hnRXfP5ScKEIqlq9Ns7LR1dFXEVADKWVGH0fjk83k=
340
+github.com/ipfs/boxo v0.36.1-0.20260204203152-f188f79fd412 h1:nfRIkMIhetCWD8jw5ya+FY+jn9ii2c+U5gdkmSS4L1Q=
341
+github.com/ipfs/boxo v0.36.1-0.20260204203152-f188f79fd412/go.mod h1:92hnRXfP5ScKEIqlq9Ns7LR1dFXEVADKWVGH0fjk83k=
342
github.com/ipfs/go-bitfield v1.1.0 h1:fh7FIo8bSwaJEh6DdTWbCeZ1eqOaOkKFI74SCnsWbGA=
343
github.com/ipfs/go-bitfield v1.1.0/go.mod h1:paqf1wjq/D2BBmzfTVFlJQ9IlFOZpg422HL0HqsGWHU=
344
github.com/ipfs/go-block-format v0.0.3/go.mod h1:4LmD4ZUw0mhO+JSKdpWwrzATiEfM7WWgQ8H5l6P8MVk=
@@ -373,8 +373,8 @@ github.com/ipfs/go-dsqueue v0.1.2 h1:jBMsgvT9Pj9l3cqI0m5jYpW/aWDYkW4Us6EuzrcSGbs
373
github.com/ipfs/go-dsqueue v0.1.2/go.mod h1:OU94YuMVUIF/ctR7Ysov9PI4gOa2XjPGN9nd8imSv78=
374
github.com/ipfs/go-fs-lock v0.1.1 h1:TecsP/Uc7WqYYatasreZQiP9EGRy4ZnKoG4yXxR33nw=
375
github.com/ipfs/go-fs-lock v0.1.1/go.mod h1:2goSXMCw7QfscHmSe09oXiR34DQeUdm+ei+dhonqly0=
376
-github.com/ipfs/go-ipfs-cmds v0.15.1-0.20260203151407-4b3827ebb483 h1:FnQqL92YxPX08/dcqE4cCSqEzwVGSdj2wprWHX+cUtM=
377
-github.com/ipfs/go-ipfs-cmds v0.15.1-0.20260203151407-4b3827ebb483/go.mod h1:YmhRbpaLKg40i9Ogj2+L41tJ+8x50fF8u1FJJD/WNhc=
376
+github.com/ipfs/go-ipfs-cmds v0.15.1-0.20260204204540-af9bcbaf5709 h1:0JiurWPnR7ZtjYW8XdfThOcOU5WlVVGQ1JY4FHHgyu8=
377
+github.com/ipfs/go-ipfs-cmds v0.15.1-0.20260204204540-af9bcbaf5709/go.mod h1:yZeTCte5zTH66bbEpLPkSog3/ImppCD00DMP7NjYmys=
378
github.com/ipfs/go-ipfs-delay v0.0.0-20181109222059-70721b86a9a8/go.mod h1:8SP1YXK1M1kXuc4KJZINY3TQQ03J2rwBG9QfXmbRPrw=
379
github.com/ipfs/go-ipfs-delay v0.0.1 h1:r/UXYyRcddO6thwOnhiznIAiSvxMECGgtv35Xs1IeRQ=
380
github.com/ipfs/go-ipfs-delay v0.0.1/go.mod h1:8SP1YXK1M1kXuc4KJZINY3TQQ03J2rwBG9QfXmbRPrw=
test/cli/add_test.go
+208
-225
@@ -8,7 +8,6 @@ import (
8
"testing"
9
"time"
10
11
- "github.com/dustin/go-humanize"
11
"github.com/ipfs/kubo/config"
12
"github.com/ipfs/kubo/test/cli/harness"
13
"github.com/ipfs/kubo/test/cli/testutils"
@@ -40,11 +39,6 @@ func TestAdd(t *testing.T) {
39
shortStringCidV1Sha512 = "bafkrgqbqt3gerhas23vuzrapkdeqf4vu2dwxp3srdj6hvg6nhsug2tgyn6mj3u23yx7utftq3i2ckw2fwdh5qmhid5qf3t35yvkc5e5ottlw6"
40
)
41
43
- const (
44
- cidV0Length = 34 // cidv0 sha2-256
45
- cidV1Length = 36 // cidv1 sha2-256
46
- )
47
-
42
t.Run("produced cid version: implicit default (CIDv0)", func(t *testing.T) {
43
t.Parallel()
44
node := harness.NewT(t).NewNode().Init().StartDaemon()
@@ -166,7 +160,7 @@ func TestAdd(t *testing.T) {
160
//
161
// UnixFSChunker=size-262144 (256KiB)
162
// Import.UnixFSFileMaxLinks=174
169
- node := harness.NewT(t).NewNode().Init("--profile=legacy-cid-v0") // legacy-cid-v0 for determinism across all params
163
+ node := harness.NewT(t).NewNode().Init("--profile=unixfs-v0-2015") // unixfs-v0-2015 for determinism across all params
164
node.UpdateConfig(func(cfg *config.Config) {
165
cfg.Import.UnixFSChunker = *config.NewOptionalString("size-262144") // 256 KiB chunks
166
cfg.Import.UnixFSFileMaxLinks = *config.NewOptionalInteger(174) // max 174 per level
@@ -187,266 +181,243 @@ func TestAdd(t *testing.T) {
181
require.Equal(t, "QmbBftNHWmjSWKLC49dMVrfnY8pjrJYntiAXirFJ7oJrNk", cidStr)
182
})
183
190
- t.Run("ipfs init --profile=legacy-cid-v0 sets config that produces legacy CIDv0", func(t *testing.T) {
191
- t.Parallel()
192
- node := harness.NewT(t).NewNode().Init("--profile=legacy-cid-v0")
193
- node.StartDaemon()
194
- defer node.StopDaemon()
184
+ // Profile-specific threshold tests are in cid_profiles_test.go (TestCIDProfiles).
185
+ // Tests here cover general ipfs add behavior not tied to specific profiles.
186
196
- cidStr := node.IPFSAddStr(shortString)
197
- require.Equal(t, shortStringCidV0, cidStr)
198
- })
199
-
200
- t.Run("ipfs init --profile=legacy-cid-v0 applies UnixFSChunker=size-262144 and UnixFSFileMaxLinks", func(t *testing.T) {
187
+ t.Run("ipfs add --hidden", func(t *testing.T) {
188
t.Parallel()
202
- seed := "v0-seed"
203
- profile := "--profile=legacy-cid-v0"
189
205
- t.Run("under UnixFSFileMaxLinks=174", func(t *testing.T) {
206
- t.Parallel()
207
- node := harness.NewT(t).NewNode().Init(profile)
208
- node.StartDaemon()
209
- defer node.StopDaemon()
210
- // Add 44544KiB file:
211
- // 174 * 256KiB should fit in single DAG layer
212
- cidStr := node.IPFSAddDeterministic("44544KiB", seed)
213
- root, err := node.InspectPBNode(cidStr)
214
- assert.NoError(t, err)
215
- require.Equal(t, 174, len(root.Links))
216
- // expect same CID every time
217
- require.Equal(t, "QmUbBALi174SnogsUzLpYbD4xPiBSFANF4iztWCsHbMKh2", cidStr)
218
- })
190
+ // Helper to create test directory with hidden file
191
+ setupTestDir := func(t *testing.T, node *harness.Node) string {
192
+ testDir, err := os.MkdirTemp(node.Dir, "hidden-test")
193
+ require.NoError(t, err)
194
+ require.NoError(t, os.WriteFile(filepath.Join(testDir, "visible.txt"), []byte("visible"), 0o644))
195
+ require.NoError(t, os.WriteFile(filepath.Join(testDir, ".hidden"), []byte("hidden"), 0o644))
196
+ return testDir
197
+ }
198
220
- t.Run("above UnixFSFileMaxLinks=174", func(t *testing.T) {
199
+ t.Run("default excludes hidden files", func(t *testing.T) {
200
t.Parallel()
222
- node := harness.NewT(t).NewNode().Init(profile)
223
- node.StartDaemon()
201
+ node := harness.NewT(t).NewNode().Init().StartDaemon()
202
defer node.StopDaemon()
225
- // add 256KiB (one more block), it should force rebalancing DAG and moving most to second layer
226
- cidStr := node.IPFSAddDeterministic("44800KiB", seed)
227
- root, err := node.InspectPBNode(cidStr)
228
- assert.NoError(t, err)
229
- require.Equal(t, 2, len(root.Links))
230
- // expect same CID every time
231
- require.Equal(t, "QmepeWtdmS1hHXx1oZXsPUv6bMrfRRKfZcoPPU4eEfjnbf", cidStr)
232
- })
233
- })
203
235
- t.Run("ipfs init --profile=legacy-cid-v0 applies UnixFSHAMTDirectoryMaxFanout=256 and UnixFSHAMTDirectorySizeThreshold=256KiB", func(t *testing.T) {
236
- t.Parallel()
237
- seed := "hamt-legacy-cid-v0"
238
- profile := "--profile=legacy-cid-v0"
204
+ testDir := setupTestDir(t, node)
205
+ cidStr := node.IPFS("add", "-r", "-Q", testDir).Stdout.Trimmed()
206
+ lsOutput := node.IPFS("ls", cidStr).Stdout.Trimmed()
207
+ require.Contains(t, lsOutput, "visible.txt")
208
+ require.NotContains(t, lsOutput, ".hidden")
209
+ })
210
240
- t.Run("under UnixFSHAMTDirectorySizeThreshold=256KiB", func(t *testing.T) {
211
+ t.Run("--hidden includes hidden files", func(t *testing.T) {
212
t.Parallel()
242
- node := harness.NewT(t).NewNode().Init(profile)
243
- node.StartDaemon()
213
+ node := harness.NewT(t).NewNode().Init().StartDaemon()
214
defer node.StopDaemon()
215
246
- randDir, err := os.MkdirTemp(node.Dir, seed)
247
- require.NoError(t, err)
248
-
249
- // Create directory with a lot of files that have filenames which together take close to UnixFSHAMTDirectorySizeThreshold in total
250
- err = createDirectoryForHAMT(randDir, cidV0Length, "255KiB", seed)
251
- require.NoError(t, err)
252
- cidStr := node.IPFS("add", "-r", "-Q", randDir).Stdout.Trimmed()
253
-
254
- // Confirm the number of links is more than UnixFSHAMTDirectorySizeThreshold (indicating regular "basic" directory"
255
- root, err := node.InspectPBNode(cidStr)
256
- assert.NoError(t, err)
257
- require.Equal(t, 903, len(root.Links))
216
+ testDir := setupTestDir(t, node)
217
+ cidStr := node.IPFS("add", "-r", "-Q", "--hidden", testDir).Stdout.Trimmed()
218
+ lsOutput := node.IPFS("ls", cidStr).Stdout.Trimmed()
219
+ require.Contains(t, lsOutput, "visible.txt")
220
+ require.Contains(t, lsOutput, ".hidden")
221
})
222
260
- t.Run("above UnixFSHAMTDirectorySizeThreshold=256KiB", func(t *testing.T) {
223
+ t.Run("-H includes hidden files", func(t *testing.T) {
224
t.Parallel()
262
- node := harness.NewT(t).NewNode().Init(profile)
263
- node.StartDaemon()
225
+ node := harness.NewT(t).NewNode().Init().StartDaemon()
226
defer node.StopDaemon()
227
266
- randDir, err := os.MkdirTemp(node.Dir, seed)
267
- require.NoError(t, err)
268
-
269
- // Create directory with a lot of files that have filenames which together take close to UnixFSHAMTDirectorySizeThreshold in total
270
- err = createDirectoryForHAMT(randDir, cidV0Length, "257KiB", seed)
271
- require.NoError(t, err)
272
- cidStr := node.IPFS("add", "-r", "-Q", randDir).Stdout.Trimmed()
273
-
274
- // Confirm this time, the number of links is less than UnixFSHAMTDirectorySizeThreshold
275
- root, err := node.InspectPBNode(cidStr)
276
- assert.NoError(t, err)
277
- require.Equal(t, 252, len(root.Links))
228
+ testDir := setupTestDir(t, node)
229
+ cidStr := node.IPFS("add", "-r", "-Q", "-H", testDir).Stdout.Trimmed()
230
+ lsOutput := node.IPFS("ls", cidStr).Stdout.Trimmed()
231
+ require.Contains(t, lsOutput, "visible.txt")
232
+ require.Contains(t, lsOutput, ".hidden")
233
})
234
})
235
281
- t.Run("ipfs init --profile=test-cid-v1 produces CIDv1 with raw leaves", func(t *testing.T) {
236
+ t.Run("ipfs add --empty-dirs", func(t *testing.T) {
237
t.Parallel()
283
- node := harness.NewT(t).NewNode().Init("--profile=test-cid-v1")
284
- node.StartDaemon()
285
- defer node.StopDaemon()
238
287
- cidStr := node.IPFSAddStr(shortString)
288
- require.Equal(t, shortStringCidV1, cidStr) // raw leaf
289
- })
239
+ // Helper to create test directory with empty subdirectory
240
+ setupTestDir := func(t *testing.T, node *harness.Node) string {
241
+ testDir, err := os.MkdirTemp(node.Dir, "empty-dirs-test")
242
+ require.NoError(t, err)
243
+ require.NoError(t, os.Mkdir(filepath.Join(testDir, "empty-subdir"), 0o755))
244
+ require.NoError(t, os.WriteFile(filepath.Join(testDir, "file.txt"), []byte("content"), 0o644))
245
+ return testDir
246
+ }
247
291
- t.Run("ipfs init --profile=test-cid-v1 applies UnixFSChunker=size-1048576", func(t *testing.T) {
292
- t.Parallel()
293
- seed := "v1-seed"
294
- profile := "--profile=test-cid-v1"
248
+ t.Run("default includes empty directories", func(t *testing.T) {
249
+ t.Parallel()
250
+ node := harness.NewT(t).NewNode().Init().StartDaemon()
251
+ defer node.StopDaemon()
252
296
- t.Run("under UnixFSFileMaxLinks=174", func(t *testing.T) {
253
+ testDir := setupTestDir(t, node)
254
+ cidStr := node.IPFS("add", "-r", "-Q", testDir).Stdout.Trimmed()
255
+ require.Contains(t, node.IPFS("ls", cidStr).Stdout.Trimmed(), "empty-subdir")
256
+ })
257
+
258
+ t.Run("--empty-dirs=true includes empty directories", func(t *testing.T) {
259
t.Parallel()
298
- node := harness.NewT(t).NewNode().Init(profile)
299
- node.StartDaemon()
260
+ node := harness.NewT(t).NewNode().Init().StartDaemon()
261
defer node.StopDaemon()
301
- // Add 174MiB file:
302
- // 174 * 1MiB should fit in single layer
303
- cidStr := node.IPFSAddDeterministic("174MiB", seed)
304
- root, err := node.InspectPBNode(cidStr)
305
- assert.NoError(t, err)
306
- require.Equal(t, 174, len(root.Links))
307
- // expect same CID every time
308
- require.Equal(t, "bafybeigwduxcf2aawppv3isnfeshnimkyplvw3hthxjhr2bdeje4tdaicu", cidStr)
262
+
263
+ testDir := setupTestDir(t, node)
264
+ cidStr := node.IPFS("add", "-r", "-Q", "--empty-dirs=true", testDir).Stdout.Trimmed()
265
+ require.Contains(t, node.IPFS("ls", cidStr).Stdout.Trimmed(), "empty-subdir")
266
})
267
311
- t.Run("above UnixFSFileMaxLinks=174", func(t *testing.T) {
268
+ t.Run("--empty-dirs=false excludes empty directories", func(t *testing.T) {
269
t.Parallel()
313
- node := harness.NewT(t).NewNode().Init(profile)
314
- node.StartDaemon()
270
+ node := harness.NewT(t).NewNode().Init().StartDaemon()
271
defer node.StopDaemon()
316
- // add +1MiB (one more block), it should force rebalancing DAG and moving most to second layer
317
- cidStr := node.IPFSAddDeterministic("175MiB", seed)
318
- root, err := node.InspectPBNode(cidStr)
319
- assert.NoError(t, err)
320
- require.Equal(t, 2, len(root.Links))
321
- // expect same CID every time
322
- require.Equal(t, "bafybeidhd7lo2n2v7lta5yamob3xwhbxcczmmtmhquwhjesi35jntf7mpu", cidStr)
272
+
273
+ testDir := setupTestDir(t, node)
274
+ cidStr := node.IPFS("add", "-r", "-Q", "--empty-dirs=false", testDir).Stdout.Trimmed()
275
+ lsOutput := node.IPFS("ls", cidStr).Stdout.Trimmed()
276
+ require.NotContains(t, lsOutput, "empty-subdir")
277
+ require.Contains(t, lsOutput, "file.txt")
278
})
279
})
280
326
- t.Run("ipfs init --profile=test-cid-v1 applies UnixFSHAMTDirectoryMaxFanout=256 and UnixFSHAMTDirectorySizeThreshold=256KiB", func(t *testing.T) {
281
+ t.Run("ipfs add symlink handling", func(t *testing.T) {
282
t.Parallel()
328
- seed := "hamt-cid-v1"
329
- profile := "--profile=test-cid-v1"
283
331
- t.Run("under UnixFSHAMTDirectorySizeThreshold=256KiB", func(t *testing.T) {
332
- t.Parallel()
333
- node := harness.NewT(t).NewNode().Init(profile)
334
- node.StartDaemon()
335
- defer node.StopDaemon()
336
-
337
- randDir, err := os.MkdirTemp(node.Dir, seed)
284
+ // Helper to create test directory structure:
285
+ // testDir/
286
+ // target.txt (file with "target content")
287
+ // link.txt -> target.txt (symlink at top level)
288
+ // subdir/
289
+ // subsubdir/
290
+ // nested-target.txt (file with "nested content")
291
+ // nested-link.txt -> nested-target.txt (symlink in sub-sub directory)
292
+ setupTestDir := func(t *testing.T, node *harness.Node) string {
293
+ testDir, err := os.MkdirTemp(node.Dir, "deref-symlinks-test")
294
require.NoError(t, err)
295
340
- // Create directory with a lot of files that have filenames which together take close to UnixFSHAMTDirectorySizeThreshold in total
341
- err = createDirectoryForHAMT(randDir, cidV1Length, "255KiB", seed)
342
- require.NoError(t, err)
343
- cidStr := node.IPFS("add", "-r", "-Q", randDir).Stdout.Trimmed()
296
+ // Top-level file and symlink
297
+ targetFile := filepath.Join(testDir, "target.txt")
298
+ require.NoError(t, os.WriteFile(targetFile, []byte("target content"), 0o644))
299
+ require.NoError(t, os.Symlink("target.txt", filepath.Join(testDir, "link.txt")))
300
345
- // Confirm the number of links is more than UnixFSHAMTDirectoryMaxFanout (indicating regular "basic" directory"
346
- root, err := node.InspectPBNode(cidStr)
347
- assert.NoError(t, err)
348
- require.Equal(t, 897, len(root.Links))
349
- })
301
+ // Nested file and symlink in sub-sub directory
302
+ subsubdir := filepath.Join(testDir, "subdir", "subsubdir")
303
+ require.NoError(t, os.MkdirAll(subsubdir, 0o755))
304
+ nestedTarget := filepath.Join(subsubdir, "nested-target.txt")
305
+ require.NoError(t, os.WriteFile(nestedTarget, []byte("nested content"), 0o644))
306
+ require.NoError(t, os.Symlink("nested-target.txt", filepath.Join(subsubdir, "nested-link.txt")))
307
+
308
+ return testDir
309
+ }
310
351
- t.Run("above UnixFSHAMTDirectorySizeThreshold=256KiB", func(t *testing.T) {
311
+ t.Run("default preserves symlinks", func(t *testing.T) {
312
t.Parallel()
353
- node := harness.NewT(t).NewNode().Init(profile)
354
- node.StartDaemon()
313
+ node := harness.NewT(t).NewNode().Init().StartDaemon()
314
defer node.StopDaemon()
315
357
- randDir, err := os.MkdirTemp(node.Dir, seed)
316
+ testDir := setupTestDir(t, node)
317
+
318
+ // Add directory with symlink (default: preserve)
319
+ dirCID := node.IPFS("add", "-r", "-Q", testDir).Stdout.Trimmed()
320
+
321
+ // Get and verify symlinks are preserved
322
+ outDir, err := os.MkdirTemp(node.Dir, "symlink-get-out")
323
require.NoError(t, err)
324
+ node.IPFS("get", "-o", outDir, dirCID)
325
360
- // Create directory with a lot of files that have filenames which together take close to UnixFSHAMTDirectorySizeThreshold in total
361
- err = createDirectoryForHAMT(randDir, cidV1Length, "257KiB", seed)
326
+ // Check top-level symlink is preserved
327
+ linkPath := filepath.Join(outDir, "link.txt")
328
+ fi, err := os.Lstat(linkPath)
329
require.NoError(t, err)
363
- cidStr := node.IPFS("add", "-r", "-Q", randDir).Stdout.Trimmed()
330
+ require.True(t, fi.Mode()&os.ModeSymlink != 0, "link.txt should be a symlink")
331
+ target, err := os.Readlink(linkPath)
332
+ require.NoError(t, err)
333
+ require.Equal(t, "target.txt", target)
334
365
- // Confirm this time, the number of links is less than UnixFSHAMTDirectoryMaxFanout
366
- root, err := node.InspectPBNode(cidStr)
367
- assert.NoError(t, err)
368
- require.Equal(t, 252, len(root.Links))
335
+ // Check nested symlink is preserved
336
+ nestedLinkPath := filepath.Join(outDir, "subdir", "subsubdir", "nested-link.txt")
337
+ fi, err = os.Lstat(nestedLinkPath)
338
+ require.NoError(t, err)
339
+ require.True(t, fi.Mode()&os.ModeSymlink != 0, "nested-link.txt should be a symlink")
340
})
370
- })
371
-
372
- t.Run("ipfs init --profile=test-cid-v1-wide applies UnixFSChunker=size-1048576 and UnixFSFileMaxLinks=1024", func(t *testing.T) {
373
- t.Parallel()
374
- seed := "v1-seed-1024"
375
- profile := "--profile=test-cid-v1-wide"
341
377
- t.Run("under UnixFSFileMaxLinks=1024", func(t *testing.T) {
342
+ // --dereference-args is deprecated but still works for backwards compatibility.
343
+ // It only resolves symlinks passed as CLI arguments, NOT symlinks found
344
+ // during directory traversal. Use --dereference-symlinks instead.
345
+ t.Run("--dereference-args resolves CLI args only", func(t *testing.T) {
346
t.Parallel()
379
- node := harness.NewT(t).NewNode().Init(profile)
380
- node.StartDaemon()
347
+ node := harness.NewT(t).NewNode().Init().StartDaemon()
348
defer node.StopDaemon()
382
- // Add 174MiB file:
383
- // 1024 * 1MiB should fit in single layer
384
- cidStr := node.IPFSAddDeterministic("1024MiB", seed)
385
- root, err := node.InspectPBNode(cidStr)
386
- assert.NoError(t, err)
387
- require.Equal(t, 1024, len(root.Links))
388
- // expect same CID every time
389
- require.Equal(t, "bafybeiej5w63ir64oxgkr5htqmlerh5k2rqflurn2howimexrlkae64xru", cidStr)
390
- })
349
392
- t.Run("above UnixFSFileMaxLinks=1024", func(t *testing.T) {
393
- t.Parallel()
394
- node := harness.NewT(t).NewNode().Init(profile)
395
- node.StartDaemon()
396
- defer node.StopDaemon()
397
- // add +1MiB (one more block), it should force rebalancing DAG and moving most to second layer
398
- cidStr := node.IPFSAddDeterministic("1025MiB", seed)
399
- root, err := node.InspectPBNode(cidStr)
400
- assert.NoError(t, err)
401
- require.Equal(t, 2, len(root.Links))
402
- // expect same CID every time
403
- require.Equal(t, "bafybeieilp2qx24pe76hxrxe6bpef5meuxto3kj5dd6mhb5kplfeglskdm", cidStr)
404
- })
405
- })
350
+ testDir := setupTestDir(t, node)
351
+ symlinkPath := filepath.Join(testDir, "link.txt")
352
+ targetPath := filepath.Join(testDir, "target.txt")
353
407
- t.Run("ipfs init --profile=test-cid-v1-wide applies UnixFSHAMTDirectoryMaxFanout=256 and UnixFSHAMTDirectorySizeThreshold=1MiB", func(t *testing.T) {
408
- t.Parallel()
409
- seed := "hamt-cid-v1"
410
- profile := "--profile=test-cid-v1-wide"
354
+ symlinkCID := node.IPFS("add", "-Q", "--dereference-args", symlinkPath).Stdout.Trimmed()
355
+ targetCID := node.IPFS("add", "-Q", targetPath).Stdout.Trimmed()
356
412
- t.Run("under UnixFSHAMTDirectorySizeThreshold=1MiB", func(t *testing.T) {
413
- t.Parallel()
414
- node := harness.NewT(t).NewNode().Init(profile)
415
- node.StartDaemon()
416
- defer node.StopDaemon()
357
+ // CIDs should match because --dereference-args resolves the symlink
358
+ require.Equal(t, targetCID, symlinkCID,
359
+ "--dereference-args should resolve CLI arg symlink to target content")
360
418
- randDir, err := os.MkdirTemp(node.Dir, seed)
419
- require.NoError(t, err)
361
+ // Now add the directory recursively with --dereference-args
362
+ // Nested symlinks should NOT be resolved (only CLI args are resolved)
363
+ dirCID := node.IPFS("add", "-r", "-Q", "--dereference-args", testDir).Stdout.Trimmed()
364
421
- // Create directory with a lot of files that have filenames which together take close to UnixFSHAMTDirectorySizeThreshold in total
422
- err = createDirectoryForHAMT(randDir, cidV1Length, "1023KiB", seed)
365
+ outDir, err := os.MkdirTemp(node.Dir, "deref-args-out")
366
require.NoError(t, err)
424
- cidStr := node.IPFS("add", "-r", "-Q", randDir).Stdout.Trimmed()
367
+ node.IPFS("get", "-o", outDir, dirCID)
368
426
- // Confirm the number of links is more than UnixFSHAMTDirectoryMaxFanout (indicating regular "basic" directory"
427
- root, err := node.InspectPBNode(cidStr)
428
- assert.NoError(t, err)
429
- require.Equal(t, 3599, len(root.Links))
369
+ // Nested symlink should still be a symlink (not dereferenced)
370
+ nestedLinkPath := filepath.Join(outDir, "subdir", "subsubdir", "nested-link.txt")
371
+ fi, err := os.Lstat(nestedLinkPath)
372
+ require.NoError(t, err)
373
+ require.True(t, fi.Mode()&os.ModeSymlink != 0,
374
+ "--dereference-args should NOT resolve nested symlinks, only CLI args")
375
})
376
432
- t.Run("above UnixFSHAMTDirectorySizeThreshold=1MiB", func(t *testing.T) {
377
+ // --dereference-symlinks resolves ALL symlinks: both CLI arguments AND
378
+ // symlinks found during directory traversal. This is a superset of
379
+ // the deprecated --dereference-args behavior.
380
+ t.Run("--dereference-symlinks resolves all symlinks", func(t *testing.T) {
381
t.Parallel()
434
- node := harness.NewT(t).NewNode().Init(profile)
435
- node.StartDaemon()
382
+ node := harness.NewT(t).NewNode().Init().StartDaemon()
383
defer node.StopDaemon()
384
438
- randDir, err := os.MkdirTemp(node.Dir, seed)
385
+ testDir := setupTestDir(t, node)
386
+ symlinkPath := filepath.Join(testDir, "link.txt")
387
+ targetPath := filepath.Join(testDir, "target.txt")
388
+
389
+ symlinkCID := node.IPFS("add", "-Q", "--dereference-symlinks", symlinkPath).Stdout.Trimmed()
390
+ targetCID := node.IPFS("add", "-Q", targetPath).Stdout.Trimmed()
391
+
392
+ require.Equal(t, targetCID, symlinkCID,
393
+ "--dereference-symlinks should resolve CLI arg symlink (like --dereference-args)")
394
+
395
+ // Test 2: Nested symlinks in sub-sub directory are ALSO resolved
396
+ dirCID := node.IPFS("add", "-r", "-Q", "--dereference-symlinks", testDir).Stdout.Trimmed()
397
+
398
+ outDir, err := os.MkdirTemp(node.Dir, "deref-symlinks-out")
399
require.NoError(t, err)
400
+ node.IPFS("get", "-o", outDir, dirCID)
401
441
- // Create directory with a lot of files that have filenames which together take close to UnixFSHAMTDirectorySizeThreshold in total
442
- err = createDirectoryForHAMT(randDir, cidV1Length, "1025KiB", seed)
402
+ // Top-level symlink should be dereferenced to regular file
403
+ linkPath := filepath.Join(outDir, "link.txt")
404
+ fi, err := os.Lstat(linkPath)
405
+ require.NoError(t, err)
406
+ require.False(t, fi.Mode()&os.ModeSymlink != 0,
407
+ "link.txt should be dereferenced to regular file")
408
+ content, err := os.ReadFile(linkPath)
409
require.NoError(t, err)
444
- cidStr := node.IPFS("add", "-r", "-Q", randDir).Stdout.Trimmed()
410
+ require.Equal(t, "target content", string(content))
411
446
- // Confirm this time, the number of links is less than UnixFSHAMTDirectoryMaxFanout
447
- root, err := node.InspectPBNode(cidStr)
448
- assert.NoError(t, err)
449
- require.Equal(t, 992, len(root.Links))
412
+ // Nested symlink in sub-sub directory should ALSO be dereferenced
413
+ nestedLinkPath := filepath.Join(outDir, "subdir", "subsubdir", "nested-link.txt")
414
+ fi, err = os.Lstat(nestedLinkPath)
415
+ require.NoError(t, err)
416
+ require.False(t, fi.Mode()&os.ModeSymlink != 0,
417
+ "nested-link.txt should be dereferenced (--dereference-symlinks resolves ALL symlinks)")
418
+ nestedContent, err := os.ReadFile(nestedLinkPath)
419
+ require.NoError(t, err)
420
+ require.Equal(t, "nested content", string(nestedContent))
421
})
422
})
423
}
@@ -627,30 +598,46 @@ func TestAddFastProvide(t *testing.T) {
598
})
599
}
600
630
-// createDirectoryForHAMT aims to create enough files with long names for the directory block to be close to the UnixFSHAMTDirectorySizeThreshold.
631
-// The calculation is based on boxo's HAMTShardingSize and sizeBelowThreshold which calculates ballpark size of the block
632
-// by adding length of link names and the binary cid length.
633
-// See https://github.com/ipfs/boxo/blob/6c5a07602aed248acc86598f30ab61923a54a83e/ipld/unixfs/io/directory.go#L491
634
-func createDirectoryForHAMT(dirPath string, cidLength int, unixfsNodeSizeTarget, seed string) error {
635
- hamtThreshold, err := humanize.ParseBytes(unixfsNodeSizeTarget)
636
- if err != nil {
637
- return err
638
- }
601
+// createDirectoryForHAMTLinksEstimation creates a directory with the specified number
602
+// of files for testing links-based size estimation (size = sum of nameLen + cidLen).
603
+// Used by legacy profiles (unixfs-v0-2015).
604
+//
605
+// The lastNameLen parameter allows the last file to have a different name length,
606
+// enabling exact +1 byte threshold tests.
607
+func createDirectoryForHAMTLinksEstimation(dirPath string, numFiles, nameLen, lastNameLen int, seed string) error {
608
+ return createDeterministicFiles(dirPath, numFiles, nameLen, lastNameLen, seed)
609
+}
610
+
611
+// createDirectoryForHAMTBlockEstimation creates a directory with the specified number
612
+// of files for testing block-based size estimation (LinkSerializedSize with protobuf overhead).
613
+// Used by modern profiles (unixfs-v1-2025).
614
+//
615
+// The lastNameLen parameter allows the last file to have a different name length,
616
+// enabling exact +1 byte threshold tests.
617
+func createDirectoryForHAMTBlockEstimation(dirPath string, numFiles, nameLen, lastNameLen int, seed string) error {
618
+ return createDeterministicFiles(dirPath, numFiles, nameLen, lastNameLen, seed)
619
+}
620
640
- // Calculate how many files with long filenames are needed to hit UnixFSHAMTDirectorySizeThreshold
641
- nameLen := 255 // max that works across windows/macos/linux
621
+// createDeterministicFiles creates numFiles files with deterministic names.
622
+// Files 0 to numFiles-2 have nameLen characters, and the last file has lastNameLen characters.
623
+// Each file contains "x" (1 byte) for non-zero tsize in directory links.
624
+func createDeterministicFiles(dirPath string, numFiles, nameLen, lastNameLen int, seed string) error {
625
alphabetLen := len(testutils.AlphabetEasy)
643
- numFiles := int(hamtThreshold) / (nameLen + cidLength)
626
645
- // Deterministic pseudo-random bytes for static CID
646
- drand, err := testutils.DeterministicRandomReader(unixfsNodeSizeTarget, seed)
627
+ // Deterministic pseudo-random bytes for static filenames
628
+ drand, err := testutils.DeterministicRandomReader("1MiB", seed)
629
if err != nil {
630
return err
631
}
632
651
- // Create necessary files in a single, flat directory
633
for i := 0; i < numFiles; i++ {
653
- buf := make([]byte, nameLen)
634
+ // Use lastNameLen for the final file
635
+ currentNameLen := nameLen
636
+ if i == numFiles-1 {
637
+ currentNameLen = lastNameLen
638
+ }
639
+
640
+ buf := make([]byte, currentNameLen)
641
_, err := io.ReadFull(drand, buf)
642
if err != nil {
643
return err
@@ -658,21 +645,17 @@ func createDirectoryForHAMT(dirPath string, cidLength int, unixfsNodeSizeTarget,
645
646
// Convert deterministic pseudo-random bytes to ASCII
647
var sb strings.Builder
661
-
648
for _, b := range buf {
663
- // Map byte to printable ASCII range (33-126)
649
char := testutils.AlphabetEasy[int(b)%alphabetLen]
650
sb.WriteRune(char)
651
}
667
- filename := sb.String()[:nameLen]
652
+ filename := sb.String()[:currentNameLen]
653
filePath := filepath.Join(dirPath, filename)
654
670
- // Create empty file
671
- f, err := os.Create(filePath)
672
- if err != nil {
655
+ // Create file with 1-byte content for non-zero tsize
656
+ if err := os.WriteFile(filePath, []byte("x"), 0o644); err != nil {
657
return err
658
}
675
- f.Close()
659
}
660
return nil
661
}
test/cli/cid_profiles_test.go
new
+724
@@ -0,0 +1,724 @@
1
+package cli
2
+
3
+import (
4
+ "encoding/json"
5
+ "os"
6
+ "path/filepath"
7
+ "strings"
8
+ "testing"
9
+
10
+ ft "github.com/ipfs/boxo/ipld/unixfs"
11
+ "github.com/ipfs/kubo/test/cli/harness"
12
+ "github.com/ipfs/kubo/test/cli/testutils"
13
+ "github.com/stretchr/testify/assert"
14
+ "github.com/stretchr/testify/require"
15
+)
16
+
17
+// cidProfileExpectations defines expected behaviors for a UnixFS import profile.
18
+// This allows DRY testing of multiple profiles with the same test logic.
19
+//
20
+// Each profile is tested against threshold boundaries to verify:
21
+// - CID format (version, hash function, raw leaves vs dag-pb wrapped)
22
+// - File chunking (UnixFSChunker size threshold)
23
+// - DAG structure (UnixFSFileMaxLinks rebalancing threshold)
24
+// - Directory sharding (HAMTThreshold for flat vs HAMT directories)
25
+type cidProfileExpectations struct {
26
+ // Profile identification
27
+ Name string // canonical profile name from IPIP-499
28
+ ProfileArgs []string // args to pass to ipfs init (empty for default behavior)
29
+
30
+ // CID format expectations
31
+ CIDVersion int // 0 or 1
32
+ HashFunc string // e.g., "sha2-256"
33
+ RawLeaves bool // true = raw codec for small files, false = dag-pb wrapped
34
+
35
+ // File chunking expectations (UnixFSChunker config)
36
+ ChunkSize int // chunk size in bytes (e.g., 262144 for 256KiB, 1048576 for 1MiB)
37
+ ChunkSizeHuman string // human-readable chunk size (e.g., "256KiB", "1MiB")
38
+ FileMaxLinks int // max links before DAG rebalancing (UnixFSFileMaxLinks config)
39
+
40
+ // HAMT directory sharding expectations (UnixFSHAMTDirectory* config).
41
+ // Threshold behavior: boxo converts to HAMT when size > HAMTThreshold (not >=).
42
+ // This means a directory exactly at the threshold stays as a basic (flat) directory.
43
+ HAMTFanout int // max links per HAMT shard bucket (256)
44
+ HAMTThreshold int // sharding threshold in bytes (262144 = 256 KiB)
45
+ HAMTSizeEstimation string // "block" (protobuf size) or "links" (legacy name+cid)
46
+
47
+ // Test vector parameters for threshold boundary tests.
48
+ // - DirBasic: size == threshold (stays basic)
49
+ // - DirHAMT: size > threshold (converts to HAMT)
50
+ // For block estimation, last filename length is adjusted to hit exact thresholds.
51
+ DirBasicNameLen int // filename length for basic directory (files 0 to N-2)
52
+ DirBasicLastNameLen int // filename length for last file (0 = same as DirBasicNameLen)
53
+ DirBasicFiles int // file count for basic directory (at exact threshold)
54
+ DirHAMTNameLen int // filename length for HAMT directory (files 0 to N-2)
55
+ DirHAMTLastNameLen int // filename length for last file (0 = same as DirHAMTNameLen)
56
+ DirHAMTFiles int // total file count for HAMT directory (over threshold)
57
+
58
+ // Expected deterministic CIDs for test vectors.
59
+ // These serve as regression tests to detect unintended changes in CID generation.
60
+
61
+ // SmallFileCID is the deterministic CID for "hello world" string.
62
+ // Tests basic CID format (version, codec, hash).
63
+ SmallFileCID string
64
+
65
+ // FileAtChunkSizeCID is the deterministic CID for a file exactly at chunk size.
66
+ // This file fits in a single block with no links:
67
+ // - v0-2015: dag-pb wrapped TFile node (CIDv0)
68
+ // - v1-2025: raw leaf block (CIDv1)
69
+ FileAtChunkSizeCID string
70
+
71
+ // FileOverChunkSizeCID is the deterministic CID for a file 1 byte over chunk size.
72
+ // This file requires 2 chunks, producing a root dag-pb node with 2 links:
73
+ // - v0-2015: links point to dag-pb wrapped TFile leaf nodes
74
+ // - v1-2025: links point to raw leaf blocks
75
+ FileOverChunkSizeCID string
76
+
77
+ // FileAtMaxLinksCID is the deterministic CID for a file at UnixFSFileMaxLinks threshold.
78
+ // File size = maxLinks * chunkSize, producing a single-layer DAG with exactly maxLinks children.
79
+ FileAtMaxLinksCID string
80
+
81
+ // FileOverMaxLinksCID is the deterministic CID for a file 1 byte over max links threshold.
82
+ // The +1 byte requires an additional chunk, forcing DAG rebalancing to 2 layers.
83
+ FileOverMaxLinksCID string
84
+
85
+ // DirBasicCID is the deterministic CID for a directory exactly at HAMTThreshold.
86
+ // With > comparison (not >=), directory at exact threshold stays as basic (flat) directory.
87
+ DirBasicCID string
88
+
89
+ // DirHAMTCID is the deterministic CID for a directory 1 byte over HAMTThreshold.
90
+ // Crossing the threshold converts the directory to a HAMT sharded structure.
91
+ DirHAMTCID string
92
+}
93
+
94
+// unixfsV02015 is the legacy profile for backward-compatible CID generation.
95
+// Alias: legacy-cid-v0
96
+var unixfsV02015 = cidProfileExpectations{
97
+ Name: "unixfs-v0-2015",
98
+ ProfileArgs: []string{"--profile=unixfs-v0-2015"},
99
+
100
+ CIDVersion: 0,
101
+ HashFunc: "sha2-256",
102
+ RawLeaves: false,
103
+
104
+ ChunkSize: 262144, // 256 KiB
105
+ ChunkSizeHuman: "256KiB",
106
+ FileMaxLinks: 174,
107
+
108
+ HAMTFanout: 256,
109
+ HAMTThreshold: 262144, // 256 KiB
110
+ HAMTSizeEstimation: "links",
111
+ DirBasicNameLen: 30, // 4096 * (30 + 34) = 262144 exactly at threshold
112
+ DirBasicFiles: 4096, // 4096 * 64 = 262144 (stays basic with >)
113
+ DirHAMTNameLen: 31, // 4033 * (31 + 34) = 262145 exactly +1 over threshold
114
+ DirHAMTLastNameLen: 0, // 0 = same as DirHAMTNameLen (uniform filenames)
115
+ DirHAMTFiles: 4033, // 4033 * 65 = 262145 (becomes HAMT)
116
+
117
+ SmallFileCID: "Qmf412jQZiuVUtdgnB36FXFX7xg5V6KEbSJ4dpQuhkLyfD", // "hello world" dag-pb wrapped
118
+ FileAtChunkSizeCID: "QmWmRj3dFDZdb6ABvbmKhEL6TmPbAfBZ1t5BxsEyJrcZhE", // 262144 bytes with seed "chunk-v0-seed"
119
+ FileOverChunkSizeCID: "QmYyLxtzZyW22zpoVAtKANLRHpDjZtNeDjQdJrcQNWoRkJ", // 262145 bytes with seed "chunk-v0-seed"
120
+ FileAtMaxLinksCID: "QmUbBALi174SnogsUzLpYbD4xPiBSFANF4iztWCsHbMKh2", // 174*256KiB bytes with seed "v0-seed"
121
+ FileOverMaxLinksCID: "QmV81WL765sC8DXsRhE5fJv2rwhS4icHRaf3J9Zk5FdRnW", // 174*256KiB+1 bytes with seed "v0-seed"
122
+ DirBasicCID: "QmX5GtRk3TSSEHtdrykgqm4eqMEn3n2XhfkFAis5fjyZmN", // 4096 files at threshold
123
+ DirHAMTCID: "QmeMiJzmhpJAUgynAcxTQYek5PPKgdv3qEvFsdV3XpVnvP", // 4033 files +1 over threshold
124
+}
125
+
126
+// unixfsV12025 is the recommended profile for cross-implementation CID determinism.
127
+var unixfsV12025 = cidProfileExpectations{
128
+ Name: "unixfs-v1-2025",
129
+ ProfileArgs: []string{"--profile=unixfs-v1-2025"},
130
+
131
+ CIDVersion: 1,
132
+ HashFunc: "sha2-256",
133
+ RawLeaves: true,
134
+
135
+ ChunkSize: 1048576, // 1 MiB
136
+ ChunkSizeHuman: "1MiB",
137
+ FileMaxLinks: 1024,
138
+
139
+ HAMTFanout: 256,
140
+ HAMTThreshold: 262144, // 256 KiB
141
+ HAMTSizeEstimation: "block",
142
+ // Block size = numFiles * linkSize + 4 bytes overhead
143
+ // LinkSerializedSize(11, 36, 1) = 55, LinkSerializedSize(21, 36, 1) = 65, LinkSerializedSize(22, 36, 1) = 66
144
+ DirBasicNameLen: 11, // 4765 files * 55 bytes
145
+ DirBasicLastNameLen: 21, // last file: 65 bytes; total: 4765*55 + 65 + 4 = 262144 (at threshold)
146
+ DirBasicFiles: 4766, // stays basic with > comparison
147
+ DirHAMTNameLen: 11, // 4765 files * 55 bytes
148
+ DirHAMTLastNameLen: 22, // last file: 66 bytes; total: 4765*55 + 66 + 4 = 262145 (+1 over threshold)
149
+ DirHAMTFiles: 4766, // becomes HAMT
150
+
151
+ SmallFileCID: "bafkreifzjut3te2nhyekklss27nh3k72ysco7y32koao5eei66wof36n5e", // "hello world" raw leaf
152
+ FileAtChunkSizeCID: "bafkreiacndfy443ter6qr2tmbbdhadvxxheowwf75s6zehscklu6ezxmta", // 1048576 bytes with seed "chunk-v1-seed"
153
+ FileOverChunkSizeCID: "bafybeigmix7t42i6jacydtquhet7srwvgpizfg7gjbq7627d35mjomtu64", // 1048577 bytes with seed "chunk-v1-seed"
154
+ FileAtMaxLinksCID: "bafybeihmf37wcuvtx4hpu7he5zl5qaf2ineo2lqlfrapokkm5zzw7zyhvm", // 1024*1MiB bytes with seed "v1-2025-seed"
155
+ FileOverMaxLinksCID: "bafybeibdsi225ugbkmpbdohnxioyab6jsqrmkts3twhpvfnzp77xtzpyhe", // 1024*1MiB+1 bytes with seed "v1-2025-seed"
156
+ DirBasicCID: "bafybeic3h7rwruealwxkacabdy45jivq2crwz6bufb5ljwupn36gicplx4", // 4766 files at 262144 bytes (threshold)
157
+ DirHAMTCID: "bafybeiegvuterwurhdtkikfhbxcldohmxp566vpjdofhzmnhv6o4freidu", // 4766 files at 262145 bytes (+1 over)
158
+}
159
+
160
+// defaultProfile points to the profile that matches Kubo's implicit default behavior.
161
+// Today this is unixfs-v0-2015. When Kubo changes defaults, update this pointer.
162
+var defaultProfile = unixfsV02015
163
+
164
+const (
165
+ cidV0Length = 34 // CIDv0 sha2-256
166
+ cidV1Length = 36 // CIDv1 sha2-256
167
+)
168
+
169
+// TestCIDProfiles generates deterministic test vectors for CID profile verification.
170
+// Set CID_PROFILES_CAR_OUTPUT environment variable to export CAR files.
171
+// Example: CID_PROFILES_CAR_OUTPUT=/tmp/cid-profiles go test -run TestCIDProfiles -v
172
+func TestCIDProfiles(t *testing.T) {
173
+ t.Parallel()
174
+
175
+ carOutputDir := os.Getenv("CID_PROFILES_CAR_OUTPUT")
176
+ exportCARs := carOutputDir != ""
177
+ if exportCARs {
178
+ if err := os.MkdirAll(carOutputDir, 0o755); err != nil {
179
+ t.Fatalf("failed to create CAR output directory: %v", err)
180
+ }
181
+ t.Logf("CAR export enabled, writing to: %s", carOutputDir)
182
+ }
183
+
184
+ // Test both IPIP-499 profiles
185
+ for _, profile := range []cidProfileExpectations{unixfsV02015, unixfsV12025} {
186
+ t.Run(profile.Name, func(t *testing.T) {
187
+ t.Parallel()
188
+ runProfileTests(t, profile, carOutputDir, exportCARs)
189
+ })
190
+ }
191
+
192
+ // Test default behavior (no profile specified)
193
+ t.Run("default", func(t *testing.T) {
194
+ t.Parallel()
195
+ // Default behavior should match defaultProfile (currently unixfs-v0-2015)
196
+ defaultExp := defaultProfile
197
+ defaultExp.Name = "default"
198
+ defaultExp.ProfileArgs = nil // no profile args = default behavior
199
+ runProfileTests(t, defaultExp, carOutputDir, exportCARs)
200
+ })
201
+}
202
+
203
+// runProfileTests runs all test vectors for a given profile.
204
+// Tests verify threshold behaviors for:
205
+// - Small files (CID format verification)
206
+// - UnixFSChunker threshold (single block vs multi-block)
207
+// - UnixFSFileMaxLinks threshold (single-layer vs rebalanced DAG)
208
+// - HAMTThreshold (basic flat directory vs HAMT sharded)
209
+func runProfileTests(t *testing.T, exp cidProfileExpectations, carOutputDir string, exportCARs bool) {
210
+ cidLen := cidV0Length
211
+ if exp.CIDVersion == 1 {
212
+ cidLen = cidV1Length
213
+ }
214
+
215
+ // Test: small file produces correct CID format
216
+ // Verifies the profile sets the expected CID version, hash function, and leaf encoding.
217
+ t.Run("small file produces correct CID format", func(t *testing.T) {
218
+ t.Parallel()
219
+ node := harness.NewT(t).NewNode().Init(exp.ProfileArgs...)
220
+ node.StartDaemon()
221
+ defer node.StopDaemon()
222
+
223
+ // Use "hello world" for determinism
224
+ cidStr := node.IPFSAddStr("hello world")
225
+
226
+ // Verify CID version (v0 starts with "Qm", v1 with "b")
227
+ verifyCIDVersion(t, node, cidStr, exp.CIDVersion)
228
+
229
+ // Verify hash function (sha2-256 for both profiles)
230
+ verifyHashFunction(t, node, cidStr, exp.HashFunc)
231
+
232
+ // Verify raw leaves vs dag-pb wrapped
233
+ // - v0-2015: dag-pb codec (wrapped)
234
+ // - v1-2025: raw codec (raw leaves)
235
+ verifyRawLeaves(t, node, cidStr, exp.RawLeaves)
236
+
237
+ // Verify deterministic CID matches expected value
238
+ if exp.SmallFileCID != "" {
239
+ require.Equal(t, exp.SmallFileCID, cidStr, "expected deterministic CID for small file")
240
+ }
241
+
242
+ if exportCARs {
243
+ carPath := filepath.Join(carOutputDir, exp.Name+"_small-file.car")
244
+ require.NoError(t, node.IPFSDagExport(cidStr, carPath))
245
+ t.Logf("exported: %s -> %s", cidStr, carPath)
246
+ }
247
+ })
248
+
249
+ // Test: file at UnixFSChunker threshold (single block)
250
+ // A file exactly at chunk size fits in one block with no links.
251
+ // - v0-2015 (256KiB): produces dag-pb wrapped TFile node
252
+ // - v1-2025 (1MiB): produces raw leaf block
253
+ t.Run("file at UnixFSChunker threshold (single block)", func(t *testing.T) {
254
+ t.Parallel()
255
+ node := harness.NewT(t).NewNode().Init(exp.ProfileArgs...)
256
+ node.StartDaemon()
257
+ defer node.StopDaemon()
258
+
259
+ // File exactly at chunk size = single block (no links)
260
+ seed := chunkSeedForProfile(exp)
261
+ cidStr := node.IPFSAddDeterministicBytes(int64(exp.ChunkSize), seed)
262
+
263
+ // Verify block structure based on raw leaves setting
264
+ if exp.RawLeaves {
265
+ // v1-2025: single block is a raw leaf (no dag-pb structure)
266
+ codec := node.IPFS("cid", "format", "-f", "%c", cidStr).Stdout.Trimmed()
267
+ require.Equal(t, "raw", codec, "single block file is raw leaf")
268
+ } else {
269
+ // v0-2015: single block is a dag-pb node with no links (TFile type)
270
+ root, err := node.InspectPBNode(cidStr)
271
+ assert.NoError(t, err)
272
+ require.Equal(t, 0, len(root.Links), "single block file has no links")
273
+ fsType, err := node.UnixFSDataType(cidStr)
274
+ require.NoError(t, err)
275
+ require.Equal(t, ft.TFile, fsType, "single block file is dag-pb wrapped (TFile)")
276
+ }
277
+
278
+ verifyHashFunction(t, node, cidStr, exp.HashFunc)
279
+
280
+ if exp.FileAtChunkSizeCID != "" {
281
+ require.Equal(t, exp.FileAtChunkSizeCID, cidStr, "expected deterministic CID for file at chunk size")
282
+ }
283
+
284
+ if exportCARs {
285
+ carPath := filepath.Join(carOutputDir, exp.Name+"_file-at-chunk-size.car")
286
+ require.NoError(t, node.IPFSDagExport(cidStr, carPath))
287
+ t.Logf("exported: %s -> %s", cidStr, carPath)
288
+ }
289
+ })
290
+
291
+ // Test: file 1 byte over UnixFSChunker threshold (2 blocks)
292
+ // A file 1 byte over chunk size requires 2 chunks.
293
+ // Root is a dag-pb node with 2 links. Leaf encoding depends on profile:
294
+ // - v0-2015: leaf blocks are dag-pb wrapped TFile nodes
295
+ // - v1-2025: leaf blocks are raw codec blocks
296
+ t.Run("file 1 byte over UnixFSChunker threshold (2 blocks)", func(t *testing.T) {
297
+ t.Parallel()
298
+ node := harness.NewT(t).NewNode().Init(exp.ProfileArgs...)
299
+ node.StartDaemon()
300
+ defer node.StopDaemon()
301
+
302
+ // File +1 byte over chunk size = 2 blocks
303
+ seed := chunkSeedForProfile(exp)
304
+ cidStr := node.IPFSAddDeterministicBytes(int64(exp.ChunkSize)+1, seed)
305
+
306
+ root, err := node.InspectPBNode(cidStr)
307
+ assert.NoError(t, err)
308
+ require.Equal(t, 2, len(root.Links), "file over chunk size has 2 links")
309
+
310
+ // Verify leaf block encoding
311
+ for _, link := range root.Links {
312
+ if exp.RawLeaves {
313
+ // v1-2025: leaves are raw blocks
314
+ leafCodec := node.IPFS("cid", "format", "-f", "%c", link.Hash.Slash).Stdout.Trimmed()
315
+ require.Equal(t, "raw", leafCodec, "leaf blocks are raw, not dag-pb")
316
+ } else {
317
+ // v0-2015: leaves are dag-pb wrapped (TFile type)
318
+ leafType, err := node.UnixFSDataType(link.Hash.Slash)
319
+ require.NoError(t, err)
320
+ require.Equal(t, ft.TFile, leafType, "leaf blocks are dag-pb wrapped (TFile)")
321
+ }
322
+ }
323
+
324
+ verifyHashFunction(t, node, cidStr, exp.HashFunc)
325
+
326
+ if exp.FileOverChunkSizeCID != "" {
327
+ require.Equal(t, exp.FileOverChunkSizeCID, cidStr, "expected deterministic CID for file over chunk size")
328
+ }
329
+
330
+ if exportCARs {
331
+ carPath := filepath.Join(carOutputDir, exp.Name+"_file-over-chunk-size.car")
332
+ require.NoError(t, node.IPFSDagExport(cidStr, carPath))
333
+ t.Logf("exported: %s -> %s", cidStr, carPath)
334
+ }
335
+ })
336
+
337
+ // Test: file at UnixFSFileMaxLinks threshold (single layer)
338
+ // A file of exactly maxLinks * chunkSize bytes fits in a single DAG layer.
339
+ // - v0-2015: 174 links (174 * 256KiB = ~44.6MiB)
340
+ // - v1-2025: 1024 links (1024 * 1MiB = 1GiB)
341
+ t.Run("file at UnixFSFileMaxLinks threshold (single layer)", func(t *testing.T) {
342
+ t.Parallel()
343
+ node := harness.NewT(t).NewNode().Init(exp.ProfileArgs...)
344
+ node.StartDaemon()
345
+ defer node.StopDaemon()
346
+
347
+ // File size = maxLinks * chunkSize (exactly at threshold)
348
+ fileSize := fileAtMaxLinksBytes(exp)
349
+ seed := seedForProfile(exp)
350
+ cidStr := node.IPFSAddDeterministicBytes(fileSize, seed)
351
+
352
+ root, err := node.InspectPBNode(cidStr)
353
+ assert.NoError(t, err)
354
+ require.Equal(t, exp.FileMaxLinks, len(root.Links),
355
+ "expected exactly %d links at max", exp.FileMaxLinks)
356
+
357
+ verifyHashFunction(t, node, cidStr, exp.HashFunc)
358
+
359
+ if exp.FileAtMaxLinksCID != "" {
360
+ require.Equal(t, exp.FileAtMaxLinksCID, cidStr, "expected deterministic CID for file at max links")
361
+ }
362
+
363
+ if exportCARs {
364
+ carPath := filepath.Join(carOutputDir, exp.Name+"_file-at-max-links.car")
365
+ require.NoError(t, node.IPFSDagExport(cidStr, carPath))
366
+ t.Logf("exported: %s -> %s", cidStr, carPath)
367
+ }
368
+ })
369
+
370
+ // Test: file 1 byte over UnixFSFileMaxLinks threshold (rebalanced DAG)
371
+ // Adding 1 byte requires an additional chunk, exceeding maxLinks.
372
+ // This triggers DAG rebalancing: chunks are grouped into intermediate nodes,
373
+ // producing a 2-layer DAG with 2 links at the root.
374
+ t.Run("file 1 byte over UnixFSFileMaxLinks threshold (rebalanced DAG)", func(t *testing.T) {
375
+ t.Parallel()
376
+ node := harness.NewT(t).NewNode().Init(exp.ProfileArgs...)
377
+ node.StartDaemon()
378
+ defer node.StopDaemon()
379
+
380
+ // +1 byte over max links threshold triggers DAG rebalancing
381
+ fileSize := fileOverMaxLinksBytes(exp)
382
+ seed := seedForProfile(exp)
383
+ cidStr := node.IPFSAddDeterministicBytes(fileSize, seed)
384
+
385
+ root, err := node.InspectPBNode(cidStr)
386
+ assert.NoError(t, err)
387
+ require.Equal(t, 2, len(root.Links), "expected 2 links after DAG rebalancing")
388
+
389
+ verifyHashFunction(t, node, cidStr, exp.HashFunc)
390
+
391
+ if exp.FileOverMaxLinksCID != "" {
392
+ require.Equal(t, exp.FileOverMaxLinksCID, cidStr, "expected deterministic CID for rebalanced file")
393
+ }
394
+
395
+ if exportCARs {
396
+ carPath := filepath.Join(carOutputDir, exp.Name+"_file-over-max-links.car")
397
+ require.NoError(t, node.IPFSDagExport(cidStr, carPath))
398
+ t.Logf("exported: %s -> %s", cidStr, carPath)
399
+ }
400
+ })
401
+
402
+ // Test: directory at HAMTThreshold (basic flat dir)
403
+ // A directory exactly at HAMTThreshold stays as a basic (flat) UnixFS directory.
404
+ // Threshold uses > comparison (not >=), so size == threshold stays basic.
405
+ // Size estimation method depends on profile:
406
+ // - v0-2015 "links": size = sum(nameLen + cidLen)
407
+ // - v1-2025 "block": size = serialized protobuf block size
408
+ t.Run("directory at HAMTThreshold (basic flat dir)", func(t *testing.T) {
409
+ t.Parallel()
410
+ node := harness.NewT(t).NewNode().Init(exp.ProfileArgs...)
411
+ node.StartDaemon()
412
+ defer node.StopDaemon()
413
+
414
+ // Use consistent seed for deterministic CIDs
415
+ seed := hamtSeedForProfile(exp)
416
+ randDir, err := os.MkdirTemp(node.Dir, seed)
417
+ require.NoError(t, err)
418
+
419
+ // Create basic (flat) directory exactly at threshold
420
+ basicLastNameLen := exp.DirBasicLastNameLen
421
+ if basicLastNameLen == 0 {
422
+ basicLastNameLen = exp.DirBasicNameLen
423
+ }
424
+ if exp.HAMTSizeEstimation == "block" {
425
+ err = createDirectoryForHAMTBlockEstimation(randDir, exp.DirBasicFiles, exp.DirBasicNameLen, basicLastNameLen, seed)
426
+ } else {
427
+ err = createDirectoryForHAMTLinksEstimation(randDir, exp.DirBasicFiles, exp.DirBasicNameLen, basicLastNameLen, seed)
428
+ }
429
+ require.NoError(t, err)
430
+
431
+ cidStr := node.IPFS("add", "-r", "-Q", randDir).Stdout.Trimmed()
432
+
433
+ // Verify UnixFS type is TDirectory (1), not THAMTShard (5)
434
+ fsType, err := node.UnixFSDataType(cidStr)
435
+ require.NoError(t, err)
436
+ require.Equal(t, ft.TDirectory, fsType, "expected basic directory (type=1) at exact threshold")
437
+
438
+ root, err := node.InspectPBNode(cidStr)
439
+ assert.NoError(t, err)
440
+ require.Equal(t, exp.DirBasicFiles, len(root.Links),
441
+ "expected basic directory with %d links", exp.DirBasicFiles)
442
+
443
+ verifyHashFunction(t, node, cidStr, exp.HashFunc)
444
+
445
+ // Verify size is exactly at threshold
446
+ if exp.HAMTSizeEstimation == "block" {
447
+ blockSize := getBlockSize(t, node, cidStr)
448
+ require.Equal(t, exp.HAMTThreshold, blockSize,
449
+ "expected basic directory block size to be exactly at threshold (%d), got %d", exp.HAMTThreshold, blockSize)
450
+ }
451
+ if exp.HAMTSizeEstimation == "links" {
452
+ linksSize := 0
453
+ for _, link := range root.Links {
454
+ linksSize += len(link.Name) + cidLen
455
+ }
456
+ require.Equal(t, exp.HAMTThreshold, linksSize,
457
+ "expected basic directory links size to be exactly at threshold (%d), got %d", exp.HAMTThreshold, linksSize)
458
+ }
459
+
460
+ if exp.DirBasicCID != "" {
461
+ require.Equal(t, exp.DirBasicCID, cidStr, "expected deterministic CID for basic directory")
462
+ }
463
+
464
+ if exportCARs {
465
+ carPath := filepath.Join(carOutputDir, exp.Name+"_dir-basic.car")
466
+ require.NoError(t, node.IPFSDagExport(cidStr, carPath))
467
+ t.Logf("exported: %s (%d files) -> %s", cidStr, exp.DirBasicFiles, carPath)
468
+ }
469
+ })
470
+
471
+ // Test: directory 1 byte over HAMTThreshold (HAMT sharded)
472
+ // A directory 1 byte over HAMTThreshold is converted to a HAMT sharded structure.
473
+ // HAMT distributes entries across buckets using consistent hashing.
474
+ // Root has at most HAMTFanout links (256), with entries distributed across buckets.
475
+ t.Run("directory 1 byte over HAMTThreshold (HAMT sharded)", func(t *testing.T) {
476
+ t.Parallel()
477
+ node := harness.NewT(t).NewNode().Init(exp.ProfileArgs...)
478
+ node.StartDaemon()
479
+ defer node.StopDaemon()
480
+
481
+ // Use consistent seed for deterministic CIDs
482
+ seed := hamtSeedForProfile(exp)
483
+ randDir, err := os.MkdirTemp(node.Dir, seed)
484
+ require.NoError(t, err)
485
+
486
+ // Create HAMT (sharded) directory exactly +1 byte over threshold
487
+ lastNameLen := exp.DirHAMTLastNameLen
488
+ if lastNameLen == 0 {
489
+ lastNameLen = exp.DirHAMTNameLen
490
+ }
491
+ if exp.HAMTSizeEstimation == "block" {
492
+ err = createDirectoryForHAMTBlockEstimation(randDir, exp.DirHAMTFiles, exp.DirHAMTNameLen, lastNameLen, seed)
493
+ } else {
494
+ err = createDirectoryForHAMTLinksEstimation(randDir, exp.DirHAMTFiles, exp.DirHAMTNameLen, lastNameLen, seed)
495
+ }
496
+ require.NoError(t, err)
497
+
498
+ cidStr := node.IPFS("add", "-r", "-Q", randDir).Stdout.Trimmed()
499
+
500
+ // Verify UnixFS type is THAMTShard (5), not TDirectory (1)
501
+ fsType, err := node.UnixFSDataType(cidStr)
502
+ require.NoError(t, err)
503
+ require.Equal(t, ft.THAMTShard, fsType, "expected HAMT directory (type=5) when over threshold")
504
+
505
+ // HAMT root has at most fanout links (actual count depends on hash distribution)
506
+ root, err := node.InspectPBNode(cidStr)
507
+ assert.NoError(t, err)
508
+ require.LessOrEqual(t, len(root.Links), exp.HAMTFanout,
509
+ "expected HAMT directory root to have <= %d links", exp.HAMTFanout)
510
+
511
+ verifyHashFunction(t, node, cidStr, exp.HashFunc)
512
+
513
+ if exp.DirHAMTCID != "" {
514
+ require.Equal(t, exp.DirHAMTCID, cidStr, "expected deterministic CID for HAMT directory")
515
+ }
516
+
517
+ if exportCARs {
518
+ carPath := filepath.Join(carOutputDir, exp.Name+"_dir-hamt.car")
519
+ require.NoError(t, node.IPFSDagExport(cidStr, carPath))
520
+ t.Logf("exported: %s (%d files, HAMT root links: %d) -> %s",
521
+ cidStr, exp.DirHAMTFiles, len(root.Links), carPath)
522
+ }
523
+ })
524
+}
525
+
526
+// verifyCIDVersion checks that the CID has the expected version.
527
+func verifyCIDVersion(t *testing.T, _ *harness.Node, cidStr string, expectedVersion int) {
528
+ t.Helper()
529
+ if expectedVersion == 0 {
530
+ require.True(t, strings.HasPrefix(cidStr, "Qm"),
531
+ "expected CIDv0 (starts with Qm), got: %s", cidStr)
532
+ } else {
533
+ require.True(t, strings.HasPrefix(cidStr, "b"),
534
+ "expected CIDv1 (base32, starts with b), got: %s", cidStr)
535
+ }
536
+}
537
+
538
+// verifyHashFunction checks that the CID uses the expected hash function.
539
+func verifyHashFunction(t *testing.T, node *harness.Node, cidStr, expectedHash string) {
540
+ t.Helper()
541
+ // Use ipfs cid format to get hash function info
542
+ // Format string %h gives the hash function name
543
+ res := node.IPFS("cid", "format", "-f", "%h", cidStr)
544
+ hashFunc := strings.TrimSpace(res.Stdout.String())
545
+ require.Equal(t, expectedHash, hashFunc,
546
+ "expected hash function %s, got %s for CID %s", expectedHash, hashFunc, cidStr)
547
+}
548
+
549
+// verifyRawLeaves checks whether the CID represents a raw leaf or dag-pb wrapped block.
550
+// For CIDv1: raw leaves have codec 0x55 (raw), wrapped have codec 0x70 (dag-pb).
551
+// For CIDv0: always dag-pb (no raw leaves possible).
552
+func verifyRawLeaves(t *testing.T, node *harness.Node, cidStr string, expectRaw bool) {
553
+ t.Helper()
554
+ // Use ipfs cid format to get codec info
555
+ // Format string %c gives the codec name
556
+ res := node.IPFS("cid", "format", "-f", "%c", cidStr)
557
+ codec := strings.TrimSpace(res.Stdout.String())
558
+
559
+ if expectRaw {
560
+ require.Equal(t, "raw", codec,
561
+ "expected raw codec for raw leaves, got %s for CID %s", codec, cidStr)
562
+ } else {
563
+ require.Equal(t, "dag-pb", codec,
564
+ "expected dag-pb codec for wrapped leaves, got %s for CID %s", codec, cidStr)
565
+ }
566
+}
567
+
568
+// getBlockSize returns the size of a block in bytes using ipfs block stat.
569
+func getBlockSize(t *testing.T, node *harness.Node, cidStr string) int {
570
+ t.Helper()
571
+ res := node.IPFS("block", "stat", "--enc=json", cidStr)
572
+ var stat struct {
573
+ Size int `json:"Size"`
574
+ }
575
+ require.NoError(t, json.Unmarshal(res.Stdout.Bytes(), &stat))
576
+ return stat.Size
577
+}
578
+
579
+// fileAtMaxLinksBytes returns the file size in bytes that produces exactly FileMaxLinks chunks.
580
+func fileAtMaxLinksBytes(exp cidProfileExpectations) int64 {
581
+ return int64(exp.FileMaxLinks) * int64(exp.ChunkSize)
582
+}
583
+
584
+// fileOverMaxLinksBytes returns the file size in bytes that triggers DAG rebalancing (+1 byte over max links threshold).
585
+func fileOverMaxLinksBytes(exp cidProfileExpectations) int64 {
586
+ return int64(exp.FileMaxLinks)*int64(exp.ChunkSize) + 1
587
+}
588
+
589
+// seedForProfile returns the deterministic seed used in add_test.go for file max links tests.
590
+func seedForProfile(exp cidProfileExpectations) string {
591
+ switch exp.Name {
592
+ case "unixfs-v0-2015", "default":
593
+ return "v0-seed"
594
+ case "unixfs-v1-2025":
595
+ return "v1-2025-seed"
596
+ default:
597
+ return exp.Name + "-seed"
598
+ }
599
+}
600
+
601
+// chunkSeedForProfile returns the deterministic seed for chunk threshold tests.
602
+func chunkSeedForProfile(exp cidProfileExpectations) string {
603
+ switch exp.Name {
604
+ case "unixfs-v0-2015", "default":
605
+ return "chunk-v0-seed"
606
+ case "unixfs-v1-2025":
607
+ return "chunk-v1-seed"
608
+ default:
609
+ return "chunk-" + exp.Name + "-seed"
610
+ }
611
+}
612
+
613
+// hamtSeedForProfile returns the deterministic seed for HAMT directory tests.
614
+// Uses the same seed for both under/at threshold tests to ensure consistency.
615
+func hamtSeedForProfile(exp cidProfileExpectations) string {
616
+ switch exp.Name {
617
+ case "unixfs-v0-2015", "default":
618
+ return "hamt-unixfs-v0-2015"
619
+ case "unixfs-v1-2025":
620
+ return "hamt-unixfs-v1-2025"
621
+ default:
622
+ return "hamt-" + exp.Name
623
+ }
624
+}
625
+
626
+// TestDefaultMatchesExpectedProfile verifies that default ipfs add behavior
627
+// matches the expected profile (currently unixfs-v0-2015).
628
+func TestDefaultMatchesExpectedProfile(t *testing.T) {
629
+ t.Parallel()
630
+
631
+ node := harness.NewT(t).NewNode().Init()
632
+ node.StartDaemon()
633
+ defer node.StopDaemon()
634
+
635
+ // Small file test
636
+ cidDefault := node.IPFSAddStr("x")
637
+
638
+ // Same file with explicit profile
639
+ nodeWithProfile := harness.NewT(t).NewNode().Init(defaultProfile.ProfileArgs...)
640
+ nodeWithProfile.StartDaemon()
641
+ defer nodeWithProfile.StopDaemon()
642
+
643
+ cidWithProfile := nodeWithProfile.IPFSAddStr("x")
644
+
645
+ require.Equal(t, cidWithProfile, cidDefault,
646
+ "default behavior should match %s profile", defaultProfile.Name)
647
+}
648
+
649
+// TestProtobufHelpers verifies the protobuf size calculation helpers.
650
+func TestProtobufHelpers(t *testing.T) {
651
+ t.Parallel()
652
+
653
+ t.Run("VarintLen", func(t *testing.T) {
654
+ // Varint encoding: 7 bits per byte, MSB indicates continuation
655
+ cases := []struct {
656
+ value uint64
657
+ expected int
658
+ }{
659
+ {0, 1},
660
+ {127, 1}, // 0x7F - max 1-byte varint
661
+ {128, 2}, // 0x80 - min 2-byte varint
662
+ {16383, 2}, // 0x3FFF - max 2-byte varint
663
+ {16384, 3}, // 0x4000 - min 3-byte varint
664
+ {2097151, 3}, // 0x1FFFFF - max 3-byte varint
665
+ {2097152, 4}, // 0x200000 - min 4-byte varint
666
+ {268435455, 4}, // 0xFFFFFFF - max 4-byte varint
667
+ {268435456, 5}, // 0x10000000 - min 5-byte varint
668
+ {34359738367, 5}, // 0x7FFFFFFFF - max 5-byte varint
669
+ }
670
+
671
+ for _, tc := range cases {
672
+ got := testutils.VarintLen(tc.value)
673
+ require.Equal(t, tc.expected, got, "VarintLen(%d)", tc.value)
674
+ }
675
+ })
676
+
677
+ t.Run("LinkSerializedSize", func(t *testing.T) {
678
+ // Test typical cases for directory links
679
+ cases := []struct {
680
+ nameLen int
681
+ cidLen int
682
+ tsize uint64
683
+ expected int
684
+ }{
685
+ // 255-char name, CIDv0 (34 bytes), tsize=0
686
+ // Inner: 1+1+34 + 1+2+255 + 1+1 = 296
687
+ // Outer: 1 + 2 + 296 = 299
688
+ {255, 34, 0, 299},
689
+ // 255-char name, CIDv1 (36 bytes), tsize=0
690
+ // Inner: 1+1+36 + 1+2+255 + 1+1 = 298
691
+ // Outer: 1 + 2 + 298 = 301
692
+ {255, 36, 0, 301},
693
+ // Short name (10 chars), CIDv1, tsize=0
694
+ // Inner: 1+1+36 + 1+1+10 + 1+1 = 52
695
+ // Outer: 1 + 1 + 52 = 54
696
+ {10, 36, 0, 54},
697
+ // 255-char name, CIDv1, large tsize
698
+ // Inner: 1+1+36 + 1+2+255 + 1+5 = 302 (tsize uses 5-byte varint)
699
+ // Outer: 1 + 2 + 302 = 305
700
+ {255, 36, 34359738367, 305},
701
+ }
702
+
703
+ for _, tc := range cases {
704
+ got := testutils.LinkSerializedSize(tc.nameLen, tc.cidLen, tc.tsize)
705
+ require.Equal(t, tc.expected, got, "LinkSerializedSize(%d, %d, %d)", tc.nameLen, tc.cidLen, tc.tsize)
706
+ }
707
+ })
708
+
709
+ t.Run("EstimateFilesForBlockThreshold", func(t *testing.T) {
710
+ threshold := 262144
711
+ nameLen := 255
712
+ cidLen := 36
713
+ var tsize uint64 = 0
714
+
715
+ numFiles := testutils.EstimateFilesForBlockThreshold(threshold, nameLen, cidLen, tsize)
716
+ require.Equal(t, 870, numFiles, "expected 870 files for threshold 262144")
717
+
718
+ numFilesUnder := testutils.EstimateFilesForBlockThreshold(threshold-1, nameLen, cidLen, tsize)
719
+ require.Equal(t, 870, numFilesUnder, "expected 870 files for threshold 262143")
720
+
721
+ numFilesOver := testutils.EstimateFilesForBlockThreshold(262185, nameLen, cidLen, tsize)
722
+ require.Equal(t, 871, numFilesOver, "expected 871 files for threshold 262185")
723
+ })
724
+}
test/cli/dag_layout_test.go
new
+147
@@ -0,0 +1,147 @@
1
+package cli
2
+
3
+import (
4
+ "os"
5
+ "path/filepath"
6
+ "strings"
7
+ "testing"
8
+
9
+ "github.com/ipfs/kubo/test/cli/harness"
10
+ "github.com/stretchr/testify/require"
11
+)
12
+
13
+// TestBalancedDAGLayout verifies that kubo uses the "balanced" DAG layout
14
+// (all leaves at same depth) rather than "balanced-packed" (varying leaf depths).
15
+//
16
+// DAG layout differences across implementations:
17
+//
18
+// - balanced: kubo, helia (all leaves at same depth, uniform traversal distance)
19
+// - balanced-packed: singularity (trailing leaves may be at different depths)
20
+// - trickle: kubo --trickle (varying depths, optimized for append-only/streaming)
21
+//
22
+// kubo does not implement balanced-packed. The trickle layout also produces
23
+// non-uniform leaf depths but with different trade-offs: trickle is optimized
24
+// for append-only and streaming reads (no seeking), while balanced-packed
25
+// minimizes node count.
26
+//
27
+// IPIP-499 documents the balanced vs balanced-packed distinction. Files larger
28
+// than dag_width × chunk_size will have different CIDs between implementations
29
+// using different layouts.
30
+//
31
+// Set DAG_LAYOUT_CAR_OUTPUT environment variable to export CAR files.
32
+// Example: DAG_LAYOUT_CAR_OUTPUT=/tmp/dag-layout go test -run TestBalancedDAGLayout -v
33
+func TestBalancedDAGLayout(t *testing.T) {
34
+ t.Parallel()
35
+
36
+ carOutputDir := os.Getenv("DAG_LAYOUT_CAR_OUTPUT")
37
+ exportCARs := carOutputDir != ""
38
+ if exportCARs {
39
+ if err := os.MkdirAll(carOutputDir, 0755); err != nil {
40
+ t.Fatalf("failed to create CAR output directory: %v", err)
41
+ }
42
+ t.Logf("CAR export enabled, writing to: %s", carOutputDir)
43
+ }
44
+
45
+ t.Run("balanced layout has uniform leaf depth", func(t *testing.T) {
46
+ t.Parallel()
47
+ node := harness.NewT(t).NewNode().Init().StartDaemon()
48
+
49
+ // Create file that triggers multi-level DAG.
50
+ // For default v0: 175 chunks × 256KiB = ~44.8 MiB (just over 174 max links)
51
+ // This creates a 2-level DAG where balanced layout ensures uniform depth.
52
+ fileSize := "45MiB"
53
+ seed := "balanced-test"
54
+
55
+ cidStr := node.IPFSAddDeterministic(fileSize, seed)
56
+
57
+ // Collect leaf depths by walking DAG
58
+ depths := collectLeafDepths(t, node, cidStr, 0)
59
+
60
+ // All leaves must be at same depth for balanced layout
61
+ require.NotEmpty(t, depths, "expected at least one leaf node")
62
+ firstDepth := depths[0]
63
+ for i, d := range depths {
64
+ require.Equal(t, firstDepth, d,
65
+ "leaf %d at depth %d, expected %d (balanced layout requires uniform leaf depth)",
66
+ i, d, firstDepth)
67
+ }
68
+ t.Logf("verified %d leaves all at depth %d (CID: %s)", len(depths), firstDepth, cidStr)
69
+
70
+ if exportCARs {
71
+ carPath := filepath.Join(carOutputDir, "balanced_"+fileSize+".car")
72
+ require.NoError(t, node.IPFSDagExport(cidStr, carPath))
73
+ t.Logf("exported: %s -> %s", cidStr, carPath)
74
+ }
75
+ })
76
+
77
+ t.Run("trickle layout has varying leaf depth", func(t *testing.T) {
78
+ t.Parallel()
79
+ node := harness.NewT(t).NewNode().Init().StartDaemon()
80
+
81
+ fileSize := "45MiB"
82
+ seed := "trickle-test"
83
+
84
+ // Add with trickle layout (--trickle flag).
85
+ // Trickle produces non-uniform leaf depths, optimized for append-only
86
+ // and streaming reads (no seeking). This subtest validates the test
87
+ // logic by confirming we can detect varying depths.
88
+ cidStr := node.IPFSAddDeterministic(fileSize, seed, "--trickle")
89
+
90
+ depths := collectLeafDepths(t, node, cidStr, 0)
91
+
92
+ // Trickle layout should have varying depths
93
+ require.NotEmpty(t, depths, "expected at least one leaf node")
94
+ minDepth, maxDepth := depths[0], depths[0]
95
+ for _, d := range depths {
96
+ if d < minDepth {
97
+ minDepth = d
98
+ }
99
+ if d > maxDepth {
100
+ maxDepth = d
101
+ }
102
+ }
103
+ require.NotEqual(t, minDepth, maxDepth,
104
+ "trickle layout should have varying leaf depths, got uniform depth %d", minDepth)
105
+ t.Logf("verified %d leaves with depths ranging from %d to %d (CID: %s)", len(depths), minDepth, maxDepth, cidStr)
106
+
107
+ if exportCARs {
108
+ carPath := filepath.Join(carOutputDir, "trickle_"+fileSize+".car")
109
+ require.NoError(t, node.IPFSDagExport(cidStr, carPath))
110
+ t.Logf("exported: %s -> %s", cidStr, carPath)
111
+ }
112
+ })
113
+}
114
+
115
+// collectLeafDepths recursively walks DAG and returns depth of each leaf node.
116
+// A node is a leaf if it's a raw block or a dag-pb node with no links.
117
+func collectLeafDepths(t *testing.T, node *harness.Node, cid string, depth int) []int {
118
+ t.Helper()
119
+
120
+ // Check codec to see if this is a raw leaf
121
+ res := node.IPFS("cid", "format", "-f", "%c", cid)
122
+ codec := strings.TrimSpace(res.Stdout.String())
123
+ if codec == "raw" {
124
+ // Raw blocks are always leaves
125
+ return []int{depth}
126
+ }
127
+
128
+ // Try to inspect as dag-pb node
129
+ pbNode, err := node.InspectPBNode(cid)
130
+ if err != nil {
131
+ // Can't parse as dag-pb, treat as leaf
132
+ return []int{depth}
133
+ }
134
+
135
+ // No links = leaf node
136
+ if len(pbNode.Links) == 0 {
137
+ return []int{depth}
138
+ }
139
+
140
+ // Recurse into children
141
+ var depths []int
142
+ for _, link := range pbNode.Links {
143
+ childDepths := collectLeafDepths(t, node, link.Hash.Slash, depth+1)
144
+ depths = append(depths, childDepths...)
145
+ }
146
+ return depths
147
+}
test/cli/files_test.go
+437
@@ -1,11 +1,14 @@
1
package cli
2
3
import (
4
+ "encoding/json"
5
"fmt"
6
"os"
7
"path/filepath"
8
+ "strings"
9
"testing"
10
11
+ ft "github.com/ipfs/boxo/ipld/unixfs"
12
"github.com/ipfs/kubo/config"
13
"github.com/ipfs/kubo/test/cli/harness"
14
"github.com/stretchr/testify/assert"
@@ -459,3 +462,437 @@ func TestFilesChroot(t *testing.T) {
462
assert.Contains(t, res.Stderr.String(), "opening repo")
463
})
464
}
465
+
466
+// TestFilesMFSImportConfig tests that MFS operations respect Import.* configuration settings.
467
+// These tests verify that `ipfs files` commands use the same import settings as `ipfs add`.
468
+func TestFilesMFSImportConfig(t *testing.T) {
469
+ t.Parallel()
470
+
471
+ t.Run("files write respects Import.CidVersion=1", func(t *testing.T) {
472
+ t.Parallel()
473
+ node := harness.NewT(t).NewNode().Init()
474
+ node.UpdateConfig(func(cfg *config.Config) {
475
+ cfg.Import.CidVersion = *config.NewOptionalInteger(1)
476
+ })
477
+ node.StartDaemon()
478
+ defer node.StopDaemon()
479
+
480
+ // Write file via MFS
481
+ tempFile := filepath.Join(node.Dir, "test.txt")
482
+ require.NoError(t, os.WriteFile(tempFile, []byte("hello"), 0644))
483
+ node.IPFS("files", "write", "--create", "/test.txt", tempFile)
484
+
485
+ // Get CID of written file
486
+ cidStr := node.IPFS("files", "stat", "--hash", "/test.txt").Stdout.Trimmed()
487
+
488
+ // Verify CIDv1 format (base32, starts with "b")
489
+ require.True(t, strings.HasPrefix(cidStr, "b"), "expected CIDv1 (starts with b), got: %s", cidStr)
490
+ })
491
+
492
+ t.Run("files write respects Import.UnixFSRawLeaves=true", func(t *testing.T) {
493
+ t.Parallel()
494
+ node := harness.NewT(t).NewNode().Init()
495
+ node.UpdateConfig(func(cfg *config.Config) {
496
+ cfg.Import.CidVersion = *config.NewOptionalInteger(1)
497
+ cfg.Import.UnixFSRawLeaves = config.True
498
+ })
499
+ node.StartDaemon()
500
+ defer node.StopDaemon()
501
+
502
+ tempFile := filepath.Join(node.Dir, "test.txt")
503
+ require.NoError(t, os.WriteFile(tempFile, []byte("hello world"), 0644))
504
+ node.IPFS("files", "write", "--create", "/test.txt", tempFile)
505
+
506
+ cidStr := node.IPFS("files", "stat", "--hash", "/test.txt").Stdout.Trimmed()
507
+ codec := node.IPFS("cid", "format", "-f", "%c", cidStr).Stdout.Trimmed()
508
+ require.Equal(t, "raw", codec, "expected raw codec for small file with raw leaves")
509
+ })
510
+
511
+ // This test verifies CID parity for single-block files only.
512
+ // Multi-block files will have different CIDs because MFS uses trickle DAG layout
513
+ // while 'ipfs add' uses balanced DAG layout. See "files write vs add for multi-block" test.
514
+ t.Run("single-block file: files write produces same CID as ipfs add", func(t *testing.T) {
515
+ t.Parallel()
516
+ node := harness.NewT(t).NewNode().Init()
517
+ node.UpdateConfig(func(cfg *config.Config) {
518
+ cfg.Import.CidVersion = *config.NewOptionalInteger(1)
519
+ cfg.Import.UnixFSRawLeaves = config.True
520
+ })
521
+ node.StartDaemon()
522
+ defer node.StopDaemon()
523
+
524
+ tempFile := filepath.Join(node.Dir, "test.txt")
525
+ require.NoError(t, os.WriteFile(tempFile, []byte("hello world"), 0644))
526
+ node.IPFS("files", "write", "--create", "/test.txt", tempFile)
527
+
528
+ mfsCid := node.IPFS("files", "stat", "--hash", "/test.txt").Stdout.Trimmed()
529
+ addCid := node.IPFSAddStr("hello world")
530
+ require.Equal(t, addCid, mfsCid, "MFS write should produce same CID as ipfs add for single-block files")
531
+ })
532
+
533
+ t.Run("files mkdir respects Import.CidVersion=1", func(t *testing.T) {
534
+ t.Parallel()
535
+ node := harness.NewT(t).NewNode().Init()
536
+ node.UpdateConfig(func(cfg *config.Config) {
537
+ cfg.Import.CidVersion = *config.NewOptionalInteger(1)
538
+ })
539
+ node.StartDaemon()
540
+ defer node.StopDaemon()
541
+
542
+ node.IPFS("files", "mkdir", "/testdir")
543
+ cidStr := node.IPFS("files", "stat", "--hash", "/testdir").Stdout.Trimmed()
544
+
545
+ // Verify CIDv1 format
546
+ require.True(t, strings.HasPrefix(cidStr, "b"), "expected CIDv1 (starts with b), got: %s", cidStr)
547
+ })
548
+
549
+ t.Run("MFS subdirectory becomes HAMT when exceeding threshold", func(t *testing.T) {
550
+ t.Parallel()
551
+ node := harness.NewT(t).NewNode().Init()
552
+ node.UpdateConfig(func(cfg *config.Config) {
553
+ // Use small threshold for faster testing
554
+ cfg.Import.UnixFSHAMTDirectorySizeThreshold = *config.NewOptionalBytes("1KiB")
555
+ cfg.Import.UnixFSHAMTDirectorySizeEstimation = *config.NewOptionalString("block")
556
+ })
557
+ node.StartDaemon()
558
+ defer node.StopDaemon()
559
+
560
+ node.IPFS("files", "mkdir", "/bigdir")
561
+
562
+ content := "x"
563
+ tempFile := filepath.Join(node.Dir, "content.txt")
564
+ require.NoError(t, os.WriteFile(tempFile, []byte(content), 0644))
565
+
566
+ // Add enough files to exceed 1KiB threshold
567
+ for i := range 25 {
568
+ node.IPFS("files", "write", "--create", fmt.Sprintf("/bigdir/file%02d", i), tempFile)
569
+ }
570
+
571
+ cidStr := node.IPFS("files", "stat", "--hash", "/bigdir").Stdout.Trimmed()
572
+ fsType, err := node.UnixFSDataType(cidStr)
573
+ require.NoError(t, err)
574
+ require.Equal(t, ft.THAMTShard, fsType, "expected HAMT directory")
575
+ })
576
+
577
+ t.Run("MFS root directory becomes HAMT when exceeding threshold", func(t *testing.T) {
578
+ t.Parallel()
579
+ node := harness.NewT(t).NewNode().Init()
580
+ node.UpdateConfig(func(cfg *config.Config) {
581
+ cfg.Import.UnixFSHAMTDirectorySizeThreshold = *config.NewOptionalBytes("1KiB")
582
+ cfg.Import.UnixFSHAMTDirectorySizeEstimation = *config.NewOptionalString("block")
583
+ })
584
+ node.StartDaemon()
585
+ defer node.StopDaemon()
586
+
587
+ content := "x"
588
+ tempFile := filepath.Join(node.Dir, "content.txt")
589
+ require.NoError(t, os.WriteFile(tempFile, []byte(content), 0644))
590
+
591
+ // Add files directly to root /
592
+ for i := range 25 {
593
+ node.IPFS("files", "write", "--create", fmt.Sprintf("/file%02d", i), tempFile)
594
+ }
595
+
596
+ cidStr := node.IPFS("files", "stat", "--hash", "/").Stdout.Trimmed()
597
+ fsType, err := node.UnixFSDataType(cidStr)
598
+ require.NoError(t, err)
599
+ require.Equal(t, ft.THAMTShard, fsType, "expected MFS root to become HAMT")
600
+ })
601
+
602
+ t.Run("MFS directory reverts from HAMT to basic when items removed", func(t *testing.T) {
603
+ t.Parallel()
604
+ node := harness.NewT(t).NewNode().Init()
605
+ node.UpdateConfig(func(cfg *config.Config) {
606
+ cfg.Import.UnixFSHAMTDirectorySizeThreshold = *config.NewOptionalBytes("1KiB")
607
+ cfg.Import.UnixFSHAMTDirectorySizeEstimation = *config.NewOptionalString("block")
608
+ })
609
+ node.StartDaemon()
610
+ defer node.StopDaemon()
611
+
612
+ node.IPFS("files", "mkdir", "/testdir")
613
+
614
+ content := "x"
615
+ tempFile := filepath.Join(node.Dir, "content.txt")
616
+ require.NoError(t, os.WriteFile(tempFile, []byte(content), 0644))
617
+
618
+ // Add files to exceed threshold
619
+ for i := range 25 {
620
+ node.IPFS("files", "write", "--create", fmt.Sprintf("/testdir/file%02d", i), tempFile)
621
+ }
622
+
623
+ // Verify it became HAMT
624
+ cidStr := node.IPFS("files", "stat", "--hash", "/testdir").Stdout.Trimmed()
625
+ fsType, err := node.UnixFSDataType(cidStr)
626
+ require.NoError(t, err)
627
+ require.Equal(t, ft.THAMTShard, fsType, "should be HAMT after adding many files")
628
+
629
+ // Remove files to get back below threshold
630
+ for i := range 20 {
631
+ node.IPFS("files", "rm", fmt.Sprintf("/testdir/file%02d", i))
632
+ }
633
+
634
+ // Verify it reverted to basic directory
635
+ cidStr = node.IPFS("files", "stat", "--hash", "/testdir").Stdout.Trimmed()
636
+ fsType, err = node.UnixFSDataType(cidStr)
637
+ require.NoError(t, err)
638
+ require.Equal(t, ft.TDirectory, fsType, "should revert to basic directory after removing files")
639
+ })
640
+
641
+ // Note: 'files write' produces DIFFERENT CIDs than 'ipfs add' for multi-block files because
642
+ // MFS uses trickle DAG layout while 'ipfs add' uses balanced DAG layout.
643
+ // Single-block files produce the same CID (tested above in "single-block file: files write...").
644
+ // For multi-block CID compatibility with 'ipfs add', use 'ipfs add --to-files' instead.
645
+
646
+ t.Run("files cp preserves original CID", func(t *testing.T) {
647
+ t.Parallel()
648
+ node := harness.NewT(t).NewNode().Init()
649
+ node.UpdateConfig(func(cfg *config.Config) {
650
+ cfg.Import.CidVersion = *config.NewOptionalInteger(1)
651
+ cfg.Import.UnixFSRawLeaves = config.True
652
+ })
653
+ node.StartDaemon()
654
+ defer node.StopDaemon()
655
+
656
+ // Add file via ipfs add
657
+ originalCid := node.IPFSAddStr("hello world")
658
+
659
+ // Copy to MFS
660
+ node.IPFS("files", "cp", fmt.Sprintf("/ipfs/%s", originalCid), "/copied.txt")
661
+
662
+ // Verify CID is preserved
663
+ mfsCid := node.IPFS("files", "stat", "--hash", "/copied.txt").Stdout.Trimmed()
664
+ require.Equal(t, originalCid, mfsCid, "files cp should preserve original CID")
665
+ })
666
+
667
+ t.Run("add --to-files respects Import config", func(t *testing.T) {
668
+ t.Parallel()
669
+ node := harness.NewT(t).NewNode().Init()
670
+ node.UpdateConfig(func(cfg *config.Config) {
671
+ cfg.Import.CidVersion = *config.NewOptionalInteger(1)
672
+ cfg.Import.UnixFSRawLeaves = config.True
673
+ })
674
+ node.StartDaemon()
675
+ defer node.StopDaemon()
676
+
677
+ // Create temp file
678
+ tempFile := filepath.Join(node.Dir, "test.txt")
679
+ require.NoError(t, os.WriteFile(tempFile, []byte("hello world"), 0644))
680
+
681
+ // Add with --to-files
682
+ addCid := node.IPFS("add", "-Q", "--to-files=/added.txt", tempFile).Stdout.Trimmed()
683
+
684
+ // Verify MFS file has same CID
685
+ mfsCid := node.IPFS("files", "stat", "--hash", "/added.txt").Stdout.Trimmed()
686
+ require.Equal(t, addCid, mfsCid)
687
+
688
+ // Should be CIDv1 raw leaf
689
+ codec := node.IPFS("cid", "format", "-f", "%c", mfsCid).Stdout.Trimmed()
690
+ require.Equal(t, "raw", codec)
691
+ })
692
+
693
+ t.Run("files mkdir respects Import.UnixFSDirectoryMaxLinks", func(t *testing.T) {
694
+ t.Parallel()
695
+ node := harness.NewT(t).NewNode().Init()
696
+ node.UpdateConfig(func(cfg *config.Config) {
697
+ cfg.Import.CidVersion = *config.NewOptionalInteger(1)
698
+ // Set low link threshold to trigger HAMT sharding at 5 links
699
+ cfg.Import.UnixFSDirectoryMaxLinks = *config.NewOptionalInteger(5)
700
+ // Also need size estimation enabled for switching to work
701
+ cfg.Import.UnixFSHAMTDirectorySizeEstimation = *config.NewOptionalString("block")
702
+ })
703
+ node.StartDaemon()
704
+ defer node.StopDaemon()
705
+
706
+ // Create directory with 6 files (exceeds max 5 links)
707
+ node.IPFS("files", "mkdir", "/testdir")
708
+
709
+ content := "x"
710
+ tempFile := filepath.Join(node.Dir, "content.txt")
711
+ require.NoError(t, os.WriteFile(tempFile, []byte(content), 0644))
712
+
713
+ for i := range 6 {
714
+ node.IPFS("files", "write", "--create", fmt.Sprintf("/testdir/file%d.txt", i), tempFile)
715
+ }
716
+
717
+ // Verify directory became HAMT sharded
718
+ cidStr := node.IPFS("files", "stat", "--hash", "/testdir").Stdout.Trimmed()
719
+ fsType, err := node.UnixFSDataType(cidStr)
720
+ require.NoError(t, err)
721
+ require.Equal(t, ft.THAMTShard, fsType, "expected HAMT directory after exceeding UnixFSDirectoryMaxLinks")
722
+ })
723
+
724
+ t.Run("files write respects Import.UnixFSChunker", func(t *testing.T) {
725
+ t.Parallel()
726
+ node := harness.NewT(t).NewNode().Init()
727
+ node.UpdateConfig(func(cfg *config.Config) {
728
+ cfg.Import.CidVersion = *config.NewOptionalInteger(1)
729
+ cfg.Import.UnixFSRawLeaves = config.True
730
+ cfg.Import.UnixFSChunker = *config.NewOptionalString("size-1024") // 1KB chunks
731
+ })
732
+ node.StartDaemon()
733
+ defer node.StopDaemon()
734
+
735
+ // Create file larger than chunk size (3KB)
736
+ data := make([]byte, 3*1024)
737
+ for i := range data {
738
+ data[i] = byte(i % 256)
739
+ }
740
+ tempFile := filepath.Join(node.Dir, "large.bin")
741
+ require.NoError(t, os.WriteFile(tempFile, data, 0644))
742
+
743
+ node.IPFS("files", "write", "--create", "/large.bin", tempFile)
744
+
745
+ // Verify chunking: 3KB file with 1KB chunks should have multiple child blocks
746
+ cidStr := node.IPFS("files", "stat", "--hash", "/large.bin").Stdout.Trimmed()
747
+ dagStatJSON := node.IPFS("dag", "stat", "--enc=json", cidStr).Stdout.Trimmed()
748
+ var dagStat struct {
749
+ UniqueBlocks int `json:"UniqueBlocks"`
750
+ }
751
+ require.NoError(t, json.Unmarshal([]byte(dagStatJSON), &dagStat))
752
+ // With 1KB chunks on a 3KB file, we expect 4 blocks (3 leaf + 1 root)
753
+ assert.Greater(t, dagStat.UniqueBlocks, 1, "expected more than 1 block with 1KB chunker on 3KB file")
754
+ })
755
+
756
+ t.Run("files write with custom chunker produces same CID as ipfs add --trickle", func(t *testing.T) {
757
+ t.Parallel()
758
+ node := harness.NewT(t).NewNode().Init()
759
+ node.UpdateConfig(func(cfg *config.Config) {
760
+ cfg.Import.CidVersion = *config.NewOptionalInteger(1)
761
+ cfg.Import.UnixFSRawLeaves = config.True
762
+ cfg.Import.UnixFSChunker = *config.NewOptionalString("size-512")
763
+ })
764
+ node.StartDaemon()
765
+ defer node.StopDaemon()
766
+
767
+ // Create test data (2KB to get multiple chunks)
768
+ data := make([]byte, 2048)
769
+ for i := range data {
770
+ data[i] = byte(i % 256)
771
+ }
772
+ tempFile := filepath.Join(node.Dir, "test.bin")
773
+ require.NoError(t, os.WriteFile(tempFile, data, 0644))
774
+
775
+ // Add via MFS
776
+ node.IPFS("files", "write", "--create", "/test.bin", tempFile)
777
+ mfsCid := node.IPFS("files", "stat", "--hash", "/test.bin").Stdout.Trimmed()
778
+
779
+ // Add via ipfs add with same chunker and trickle (MFS always uses trickle)
780
+ addCid := node.IPFS("add", "-Q", "--chunker=size-512", "--trickle", tempFile).Stdout.Trimmed()
781
+
782
+ // CIDs should match when using same chunker + trickle layout
783
+ require.Equal(t, addCid, mfsCid, "MFS and add --trickle should produce same CID with matching chunker")
784
+ })
785
+
786
+ t.Run("files mkdir respects Import.UnixFSHAMTDirectoryMaxFanout", func(t *testing.T) {
787
+ t.Parallel()
788
+ node := harness.NewT(t).NewNode().Init()
789
+ node.UpdateConfig(func(cfg *config.Config) {
790
+ // Use non-default fanout of 64 (default is 256)
791
+ cfg.Import.UnixFSHAMTDirectoryMaxFanout = *config.NewOptionalInteger(64)
792
+ // Set low link threshold to trigger HAMT at 5 links
793
+ cfg.Import.UnixFSDirectoryMaxLinks = *config.NewOptionalInteger(5)
794
+ cfg.Import.UnixFSHAMTDirectorySizeEstimation = *config.NewOptionalString("disabled")
795
+ })
796
+ node.StartDaemon()
797
+ defer node.StopDaemon()
798
+
799
+ node.IPFS("files", "mkdir", "/testdir")
800
+
801
+ content := "x"
802
+ tempFile := filepath.Join(node.Dir, "content.txt")
803
+ require.NoError(t, os.WriteFile(tempFile, []byte(content), 0644))
804
+
805
+ // Add 6 files (exceeds MaxLinks=5) to trigger HAMT
806
+ for i := range 6 {
807
+ node.IPFS("files", "write", "--create", fmt.Sprintf("/testdir/file%d.txt", i), tempFile)
808
+ }
809
+
810
+ // Verify directory became HAMT
811
+ cidStr := node.IPFS("files", "stat", "--hash", "/testdir").Stdout.Trimmed()
812
+ fsType, err := node.UnixFSDataType(cidStr)
813
+ require.NoError(t, err)
814
+ require.Equal(t, ft.THAMTShard, fsType, "expected HAMT directory")
815
+
816
+ // Verify the HAMT uses the custom fanout (64) by inspecting the UnixFS Data field.
817
+ fanout, err := node.UnixFSHAMTFanout(cidStr)
818
+ require.NoError(t, err)
819
+ require.Equal(t, uint64(64), fanout, "expected HAMT fanout 64")
820
+ })
821
+
822
+ t.Run("files mkdir respects Import.UnixFSHAMTDirectorySizeThreshold", func(t *testing.T) {
823
+ t.Parallel()
824
+ node := harness.NewT(t).NewNode().Init()
825
+ node.UpdateConfig(func(cfg *config.Config) {
826
+ // Use very small threshold (100 bytes) to trigger HAMT quickly
827
+ cfg.Import.UnixFSHAMTDirectorySizeThreshold = *config.NewOptionalBytes("100B")
828
+ cfg.Import.UnixFSHAMTDirectorySizeEstimation = *config.NewOptionalString("block")
829
+ })
830
+ node.StartDaemon()
831
+ defer node.StopDaemon()
832
+
833
+ node.IPFS("files", "mkdir", "/testdir")
834
+
835
+ content := "test content"
836
+ tempFile := filepath.Join(node.Dir, "content.txt")
837
+ require.NoError(t, os.WriteFile(tempFile, []byte(content), 0644))
838
+
839
+ // Add 3 files - each link adds ~40-50 bytes, so 3 should exceed 100B threshold
840
+ for i := range 3 {
841
+ node.IPFS("files", "write", "--create", fmt.Sprintf("/testdir/file%d.txt", i), tempFile)
842
+ }
843
+
844
+ // Verify directory became HAMT due to size threshold
845
+ cidStr := node.IPFS("files", "stat", "--hash", "/testdir").Stdout.Trimmed()
846
+ fsType, err := node.UnixFSDataType(cidStr)
847
+ require.NoError(t, err)
848
+ require.Equal(t, ft.THAMTShard, fsType, "expected HAMT directory after exceeding size threshold")
849
+ })
850
+
851
+ t.Run("config change takes effect after daemon restart", func(t *testing.T) {
852
+ t.Parallel()
853
+ node := harness.NewT(t).NewNode().Init()
854
+
855
+ // Start with high threshold (won't trigger HAMT)
856
+ node.UpdateConfig(func(cfg *config.Config) {
857
+ cfg.Import.UnixFSHAMTDirectorySizeThreshold = *config.NewOptionalBytes("256KiB")
858
+ cfg.Import.UnixFSHAMTDirectorySizeEstimation = *config.NewOptionalString("block")
859
+ })
860
+ node.StartDaemon()
861
+
862
+ // Create directory with some files
863
+ node.IPFS("files", "mkdir", "/testdir")
864
+ content := "test"
865
+ tempFile := filepath.Join(node.Dir, "content.txt")
866
+ require.NoError(t, os.WriteFile(tempFile, []byte(content), 0644))
867
+ for i := range 3 {
868
+ node.IPFS("files", "write", "--create", fmt.Sprintf("/testdir/file%d.txt", i), tempFile)
869
+ }
870
+
871
+ // Verify it's still a basic directory (threshold not exceeded)
872
+ cidStr := node.IPFS("files", "stat", "--hash", "/testdir").Stdout.Trimmed()
873
+ fsType, err := node.UnixFSDataType(cidStr)
874
+ require.NoError(t, err)
875
+ require.Equal(t, ft.TDirectory, fsType, "should be basic directory with high threshold")
876
+
877
+ // Stop daemon
878
+ node.StopDaemon()
879
+
880
+ // Change config to use very low threshold
881
+ node.UpdateConfig(func(cfg *config.Config) {
882
+ cfg.Import.UnixFSHAMTDirectorySizeThreshold = *config.NewOptionalBytes("100B")
883
+ })
884
+
885
+ // Restart daemon
886
+ node.StartDaemon()
887
+ defer node.StopDaemon()
888
+
889
+ // Add one more file - this should trigger HAMT conversion with new threshold
890
+ node.IPFS("files", "write", "--create", "/testdir/file3.txt", tempFile)
891
+
892
+ // Verify it became HAMT (new threshold applied)
893
+ cidStr = node.IPFS("files", "stat", "--hash", "/testdir").Stdout.Trimmed()
894
+ fsType, err = node.UnixFSDataType(cidStr)
895
+ require.NoError(t, err)
896
+ require.Equal(t, ft.THAMTShard, fsType, "should be HAMT after daemon restart with lower threshold")
897
+ })
898
+}
test/cli/harness/ipfs.go
+23
-5
@@ -4,6 +4,7 @@ import (
4
"encoding/json"
5
"fmt"
6
"io"
7
+ "os"
8
"reflect"
9
"strings"
10
@@ -76,7 +77,8 @@ func (n *Node) IPFSAddStr(content string, args ...string) string {
77
return n.IPFSAdd(strings.NewReader(content), args...)
78
}
79
79
-// IPFSAddDeterministic produces a CID of a file of a certain size, filled with deterministically generated bytes based on some seed.
80
+// IPFSAddDeterministic produces a CID of a file of a certain size, filled with deterministically generated bytes based on some seed.
81
+// Size is specified as a humanize string (e.g., "256KiB", "1MiB").
82
// This ensures deterministic CID on the other end, that can be used in tests.
83
func (n *Node) IPFSAddDeterministic(size string, seed string, args ...string) string {
84
log.Debugf("node %d adding %s of deterministic pseudo-random data with seed %q and args: %v", n.ID, size, seed, args)
@@ -87,6 +89,17 @@ func (n *Node) IPFSAddDeterministic(size string, seed string, args ...string) st
89
return n.IPFSAdd(reader, args...)
90
}
91
92
+// IPFSAddDeterministicBytes produces a CID of a file of exactly `size` bytes, filled with deterministically generated bytes based on some seed.
93
+// Use this when exact byte precision is needed (e.g., threshold tests at T and T+1 bytes).
94
+func (n *Node) IPFSAddDeterministicBytes(size int64, seed string, args ...string) string {
95
+ log.Debugf("node %d adding %d bytes of deterministic pseudo-random data with seed %q and args: %v", n.ID, size, seed, args)
96
+ reader, err := DeterministicRandomReaderBytes(size, seed)
97
+ if err != nil {
98
+ panic(err)
99
+ }
100
+ return n.IPFSAdd(reader, args...)
101
+}
102
+
103
func (n *Node) IPFSAdd(content io.Reader, args ...string) string {
104
log.Debugf("node %d adding with args: %v", n.ID, args)
105
fullArgs := []string{"add", "-q"}
@@ -148,9 +161,15 @@ func (n *Node) IPFSDagImport(content io.Reader, cid string, args ...string) erro
161
return res.Err
162
}
163
151
-/*
152
-func (n *Node) IPFSDagExport(cid string, car *os.File) error {
153
- log.Debugf("node %d dag export of %s to %q with args: %v", n.ID, cid, car.Name())
164
+// IPFSDagExport exports a DAG rooted at cid to a CAR file at carPath.
165
+func (n *Node) IPFSDagExport(cid string, carPath string) error {
166
+ log.Debugf("node %d dag export of %s to %q", n.ID, cid, carPath)
167
+ car, err := os.Create(carPath)
168
+ if err != nil {
169
+ return err
170
+ }
171
+ defer car.Close()
172
+
173
res := n.Runner.MustRun(RunRequest{
174
Path: n.IPFSBin,
175
Args: []string{"dag", "export", cid},
@@ -158,4 +177,3 @@ func (n *Node) IPFSDagExport(cid string, car *os.File) error {
177
})
178
return res.Err
179
}
161
-*/
test/cli/harness/pbinspect.go
+69
-1
@@ -3,8 +3,77 @@ package harness
3
import (
4
"bytes"
5
"encoding/json"
6
+
7
+ mdag "github.com/ipfs/boxo/ipld/merkledag"
8
+ ft "github.com/ipfs/boxo/ipld/unixfs"
9
+ pb "github.com/ipfs/boxo/ipld/unixfs/pb"
10
)
11
12
+// UnixFSDataType returns the UnixFS DataType for the given CID by fetching the
13
+// raw block and parsing the protobuf. This directly checks the Type field in
14
+// the UnixFS Data message (https://specs.ipfs.tech/unixfs/#data).
15
+//
16
+// Common types:
17
+// - ft.TDirectory (1) = basic flat directory
18
+// - ft.THAMTShard (5) = HAMT sharded directory
19
+func (n *Node) UnixFSDataType(cid string) (pb.Data_DataType, error) {
20
+ log.Debugf("node %d block get %s", n.ID, cid)
21
+
22
+ var blockData bytes.Buffer
23
+ res := n.Runner.MustRun(RunRequest{
24
+ Path: n.IPFSBin,
25
+ Args: []string{"block", "get", cid},
26
+ CmdOpts: []CmdOpt{RunWithStdout(&blockData)},
27
+ })
28
+ if res.Err != nil {
29
+ return 0, res.Err
30
+ }
31
+
32
+ // Parse dag-pb block
33
+ protoNode, err := mdag.DecodeProtobuf(blockData.Bytes())
34
+ if err != nil {
35
+ return 0, err
36
+ }
37
+
38
+ // Parse UnixFS data
39
+ fsNode, err := ft.FSNodeFromBytes(protoNode.Data())
40
+ if err != nil {
41
+ return 0, err
42
+ }
43
+
44
+ return fsNode.Type(), nil
45
+}
46
+
47
+// UnixFSHAMTFanout returns the fanout value for a HAMT shard directory.
48
+// This is only valid for HAMT shards (THAMTShard type).
49
+func (n *Node) UnixFSHAMTFanout(cid string) (uint64, error) {
50
+ log.Debugf("node %d block get %s for fanout", n.ID, cid)
51
+
52
+ var blockData bytes.Buffer
53
+ res := n.Runner.MustRun(RunRequest{
54
+ Path: n.IPFSBin,
55
+ Args: []string{"block", "get", cid},
56
+ CmdOpts: []CmdOpt{RunWithStdout(&blockData)},
57
+ })
58
+ if res.Err != nil {
59
+ return 0, res.Err
60
+ }
61
+
62
+ // Parse dag-pb block
63
+ protoNode, err := mdag.DecodeProtobuf(blockData.Bytes())
64
+ if err != nil {
65
+ return 0, err
66
+ }
67
+
68
+ // Parse UnixFS data
69
+ fsNode, err := ft.FSNodeFromBytes(protoNode.Data())
70
+ if err != nil {
71
+ return 0, err
72
+ }
73
+
74
+ return fsNode.Fanout(), nil
75
+}
76
+
77
// InspectPBNode uses dag-json output of 'ipfs dag get' to inspect
78
// "Logical Format" of DAG-PB as defined in
79
// https://web.archive.org/web/20250403194752/https://ipld.io/specs/codecs/dag-pb/spec/#logical-format
@@ -28,7 +97,6 @@ func (n *Node) InspectPBNode(cid string) (PBNode, error) {
97
return root, err
98
}
99
return root, nil
31
-
100
}
101
102
// Define structs to match the JSON for
test/cli/testutils/protobuf.go
new
+39
@@ -0,0 +1,39 @@
1
+package testutils
2
+
3
+import "math/bits"
4
+
5
+// VarintLen returns the number of bytes needed to encode v as a protobuf varint.
6
+func VarintLen(v uint64) int {
7
+ return int(9*uint32(bits.Len64(v))+64) / 64
8
+}
9
+
10
+// LinkSerializedSize calculates the serialized size of a single PBLink in a dag-pb block.
11
+// This matches the calculation in boxo/ipld/unixfs/io/directory.go estimatedBlockSize().
12
+//
13
+// The protobuf wire format for a PBLink is:
14
+//
15
+// PBNode.Links wrapper tag (1 byte)
16
+// + varint length of inner message
17
+// + Hash field: tag (1) + varint(cidLen) + cidLen
18
+// + Name field: tag (1) + varint(nameLen) + nameLen
19
+// + Tsize field: tag (1) + varint(tsize)
20
+func LinkSerializedSize(nameLen, cidLen int, tsize uint64) int {
21
+ // Inner link message size
22
+ linkLen := 1 + VarintLen(uint64(cidLen)) + cidLen + // Hash field
23
+ 1 + VarintLen(uint64(nameLen)) + nameLen + // Name field
24
+ 1 + VarintLen(tsize) // Tsize field
25
+
26
+ // Outer wrapper: tag (1 byte) + varint(linkLen) + linkLen
27
+ return 1 + VarintLen(uint64(linkLen)) + linkLen
28
+}
29
+
30
+// EstimateFilesForBlockThreshold estimates how many files with given name/cid lengths
31
+// will fit under the block size threshold.
32
+// Returns the number of files that keeps the block size just under the threshold.
33
+func EstimateFilesForBlockThreshold(threshold, nameLen, cidLen int, tsize uint64) int {
34
+ linkSize := LinkSerializedSize(nameLen, cidLen, tsize)
35
+ // Base overhead for empty directory node (Data field + minimal structure)
36
+ // Empirically determined to be 4 bytes for dag-pb directories
37
+ baseOverhead := 4
38
+ return (threshold - baseOverhead) / linkSize
39
+}
test/cli/testutils/random_deterministic.go
+9
-3
@@ -27,13 +27,19 @@ func (r *randomReader) Read(p []byte) (int, error) {
27
return int(n), nil
28
}
29
30
-// createRandomReader produces specified number of pseudo-random bytes
31
-// from a seed.
30
+// DeterministicRandomReader produces specified number of pseudo-random bytes
31
+// from a seed. Size can be specified as a humanize string (e.g., "256KiB", "1MiB").
32
func DeterministicRandomReader(sizeStr string, seed string) (io.Reader, error) {
33
size, err := humanize.ParseBytes(sizeStr)
34
if err != nil {
35
return nil, err
36
}
37
+ return DeterministicRandomReaderBytes(int64(size), seed)
38
+}
39
+
40
+// DeterministicRandomReaderBytes produces exactly `size` pseudo-random bytes
41
+// from a seed. Use this when exact byte precision is needed.
42
+func DeterministicRandomReaderBytes(size int64, seed string) (io.Reader, error) {
43
// Hash the seed string to a 32-byte key for ChaCha20
44
key := sha256.Sum256([]byte(seed))
45
// Use ChaCha20 for deterministic random bytes
@@ -42,5 +48,5 @@ func DeterministicRandomReader(sizeStr string, seed string) (io.Reader, error) {
48
if err != nil {
49
return nil, err
50
}
45
- return &randomReader{cipher: cipher, remaining: int64(size)}, nil
51
+ return &randomReader{cipher: cipher, remaining: size}, nil
52
}
test/dependencies/go.mod
+2
-2
@@ -135,13 +135,13 @@ require (
135
github.com/huin/goupnp v1.3.0 // indirect
136
github.com/inconshreveable/mousetrap v1.1.0 // indirect
137
github.com/ipfs/bbloom v0.0.4 // indirect
138
- github.com/ipfs/boxo v0.36.1-0.20260204011824-2688767ff981 // indirect
138
+ github.com/ipfs/boxo v0.36.1-0.20260204203152-f188f79fd412 // indirect
139
github.com/ipfs/go-bitfield v1.1.0 // indirect
140
github.com/ipfs/go-block-format v0.2.3 // indirect
141
github.com/ipfs/go-cid v0.6.0 // indirect
142
github.com/ipfs/go-datastore v0.9.0 // indirect
143
github.com/ipfs/go-dsqueue v0.1.2 // indirect
144
- github.com/ipfs/go-ipfs-cmds v0.15.1-0.20260203151407-4b3827ebb483 // indirect
144
+ github.com/ipfs/go-ipfs-cmds v0.15.1-0.20260204204540-af9bcbaf5709 // indirect
145
github.com/ipfs/go-ipfs-redirects-file v0.1.2 // indirect
146
github.com/ipfs/go-ipld-cbor v0.2.1 // indirect
147
github.com/ipfs/go-ipld-format v0.6.3 // indirect
test/dependencies/go.sum
+4
-4
@@ -296,8 +296,8 @@ github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2
296
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
297
github.com/ipfs/bbloom v0.0.4 h1:Gi+8EGJ2y5qiD5FbsbpX/TMNcJw8gSqr7eyjHa4Fhvs=
298
github.com/ipfs/bbloom v0.0.4/go.mod h1:cS9YprKXpoZ9lT0n/Mw/a6/aFV6DTjTLYHeA+gyqMG0=
299
-github.com/ipfs/boxo v0.36.1-0.20260204011824-2688767ff981 h1:Q3XjjicNTpok8gD0WwbLYZpmbRoykNTiCLbpj3EjnPc=
300
-github.com/ipfs/boxo v0.36.1-0.20260204011824-2688767ff981/go.mod h1:92hnRXfP5ScKEIqlq9Ns7LR1dFXEVADKWVGH0fjk83k=
299
+github.com/ipfs/boxo v0.36.1-0.20260204203152-f188f79fd412 h1:nfRIkMIhetCWD8jw5ya+FY+jn9ii2c+U5gdkmSS4L1Q=
300
+github.com/ipfs/boxo v0.36.1-0.20260204203152-f188f79fd412/go.mod h1:92hnRXfP5ScKEIqlq9Ns7LR1dFXEVADKWVGH0fjk83k=
301
github.com/ipfs/go-bitfield v1.1.0 h1:fh7FIo8bSwaJEh6DdTWbCeZ1eqOaOkKFI74SCnsWbGA=
302
github.com/ipfs/go-bitfield v1.1.0/go.mod h1:paqf1wjq/D2BBmzfTVFlJQ9IlFOZpg422HL0HqsGWHU=
303
github.com/ipfs/go-block-format v0.2.3 h1:mpCuDaNXJ4wrBJLrtEaGFGXkferrw5eqVvzaHhtFKQk=
@@ -314,8 +314,8 @@ github.com/ipfs/go-ds-leveldb v0.5.2 h1:6nmxlQ2zbp4LCNdJVsmHfs9GP0eylfBNxpmY1csp
314
github.com/ipfs/go-ds-leveldb v0.5.2/go.mod h1:2fAwmcvD3WoRT72PzEekHBkQmBDhc39DJGoREiuGmYo=
315
github.com/ipfs/go-dsqueue v0.1.2 h1:jBMsgvT9Pj9l3cqI0m5jYpW/aWDYkW4Us6EuzrcSGbs=
316
github.com/ipfs/go-dsqueue v0.1.2/go.mod h1:OU94YuMVUIF/ctR7Ysov9PI4gOa2XjPGN9nd8imSv78=
317
-github.com/ipfs/go-ipfs-cmds v0.15.1-0.20260203151407-4b3827ebb483 h1:FnQqL92YxPX08/dcqE4cCSqEzwVGSdj2wprWHX+cUtM=
318
-github.com/ipfs/go-ipfs-cmds v0.15.1-0.20260203151407-4b3827ebb483/go.mod h1:YmhRbpaLKg40i9Ogj2+L41tJ+8x50fF8u1FJJD/WNhc=
317
+github.com/ipfs/go-ipfs-cmds v0.15.1-0.20260204204540-af9bcbaf5709 h1:0JiurWPnR7ZtjYW8XdfThOcOU5WlVVGQ1JY4FHHgyu8=
318
+github.com/ipfs/go-ipfs-cmds v0.15.1-0.20260204204540-af9bcbaf5709/go.mod h1:yZeTCte5zTH66bbEpLPkSog3/ImppCD00DMP7NjYmys=
319
github.com/ipfs/go-ipfs-delay v0.0.1 h1:r/UXYyRcddO6thwOnhiznIAiSvxMECGgtv35Xs1IeRQ=
320
github.com/ipfs/go-ipfs-delay v0.0.1/go.mod h1:8SP1YXK1M1kXuc4KJZINY3TQQ03J2rwBG9QfXmbRPrw=
321
github.com/ipfs/go-ipfs-pq v0.0.4 h1:U7jjENWJd1jhcrR8X/xHTaph14PTAK9O+yaLJbjqgOw=
test/sharness/t0250-files-api.sh
+20
-11
@@ -786,6 +786,7 @@ tests_for_files_api() {
786
test_expect_success "can create some files for testing ($EXTRA)" '
787
create_files
788
'
789
+ # default: CIDv0, dag-pb for all files (no raw-leaves)
790
ROOT_HASH=QmcwKfTMCT7AaeiD92hWjnZn9b6eh9NxnhfSzN5x2vnDpt
791
CATS_HASH=Qma88m8ErTGkZHbBWGqy1C7VmEmX8wwNDWNpGyCaNmEgwC
792
FILE_HASH=QmQdQt9qooenjeaNhiKHF3hBvmNteB4MQBtgu3jxgf9c7i
@@ -796,20 +797,23 @@ tests_for_files_api() {
797
create_files --raw-leaves
798
'
799
800
+ # partial raw-leaves: initial files created with --raw-leaves, test ops without
801
if [ "$EXTRA" = "with-daemon" ]; then
802
ROOT_HASH=QmTpKiKcAj4sbeesN6vrs5w3QeVmd4QmGpxRL81hHut4dZ
803
CATS_HASH=QmPhPkmtUGGi8ySPHoPu1qbfryLJKKq1GYxpgLyyCruvGe
804
test_files_api "($EXTRA, partial raw-leaves)"
805
fi
806
805
- ROOT_HASH=QmW3dMSU6VNd1mEdpk9S3ZYRuR1YwwoXjGaZhkyK6ru9YU
806
- CATS_HASH=QmPqWDEg7NoWRX8Y4vvYjZtmdg5umbfsTQ9zwNr12JoLmt
807
- FILE_HASH=QmRCgHeoKxCqK2Es6M6nPUDVWz19yNQPnsXGsXeuTkSKpN
808
- TRUNC_HASH=QmckstrVxJuecVD1FHUiURJiU9aPURZWJieeBVHJPACj8L
807
+ # raw-leaves: single-block files become RawNode (CIDv1), dirs stay CIDv0
808
+ ROOT_HASH=QmTHzLiSouBHVTssS8xRzmfWGAvTGhPEjtPdB6pWMQdxJX
809
+ CATS_HASH=QmPJkzbCoBuL379TbHgwF1YbVHnKgiDa5bjqYhe6Lovdms
810
+ FILE_HASH=bafybeibkrazpbejqh3qun7xfnsl7yofl74o4jwhxebpmtrcpavebokuqtm
811
+ TRUNC_HASH=bafybeigwhb3q36yrm37jv5fo2ap6r6eyohckqrxmlejrenex4xlnuxiy3e
812
test_files_api "($EXTRA, raw-leaves)" '' --raw-leaves
813
811
- ROOT_HASH=QmageRWxC7wWjPv5p36NeAgBAiFdBHaNfxAehBSwzNech2
812
- CATS_HASH=bafybeig4cpvfu2qwwo3u4ffazhqdhyynfhnxqkzvbhrdbamauthf5mfpuq
814
+ # cidv1 for mkdir: different from raw-leaves since mkdir forces CIDv1 dirs
815
+ ROOT_HASH=QmTLdTaZNj8Mvq1cgYup59ZFJFv1KxptouFSZUZKeq7X3z
816
+ CATS_HASH=bafybeihsqinttigpskqqj63wgalrny3lifvqv5ml7igrirdhlcf73l3wvm
817
FILE_HASH=bafybeibkrazpbejqh3qun7xfnsl7yofl74o4jwhxebpmtrcpavebokuqtm
818
TRUNC_HASH=bafybeigwhb3q36yrm37jv5fo2ap6r6eyohckqrxmlejrenex4xlnuxiy3e
819
if [ "$EXTRA" = "with-daemon" ]; then
@@ -823,8 +827,10 @@ tests_for_files_api() {
827
test_cmp hash_expect hash_actual
828
'
829
826
- ROOT_HASH=bafybeifxnoetaa2jetwmxubv3gqiyaknnujwkkkhdeua63kulm63dcr5wu
827
- test_files_api "($EXTRA, cidv1 root)"
830
+ # cidv1 root: root upgraded to CIDv1 via chcid, all new dirs/files also CIDv1
831
+ ROOT_HASH=bafybeickjecu37qv6ue54ofk3n4rpm4g4abuofz7yc4qn4skffy263kkou
832
+ CATS_HASH=bafybeihsqinttigpskqqj63wgalrny3lifvqv5ml7igrirdhlcf73l3wvm
833
+ test_files_api "($EXTRA, cidv1 root)"
834
835
if [ "$EXTRA" = "with-daemon" ]; then
836
test_expect_success "can update root hash to blake2b-256" '
@@ -833,8 +839,9 @@ tests_for_files_api() {
839
ipfs files stat --hash / > hash_actual &&
840
test_cmp hash_expect hash_actual
841
'
836
- ROOT_HASH=bafykbzaceb6jv27itwfun6wsrbaxahpqthh5be2bllsjtb3qpmly3vji4mlfk
837
- CATS_HASH=bafykbzacebhpn7rtcjjc5oa4zgzivhs7a6e2tq4uk4px42bubnmhpndhqtjig
842
+ # blake2b-256 root: using blake2b-256 hash instead of sha2-256
843
+ ROOT_HASH=bafykbzaceaebvwrjdw5rfhqqh5miaq3g42yybnrw3kxxxx43ggyttm6xn2zek
844
+ CATS_HASH=bafykbzaceaqvpxs3dfl7su6744jgyvifbusow2tfixdy646chasdwyz2boagc
845
FILE_HASH=bafykbzaceca45w2i3o3q3ctqsezdv5koakz7sxsw37ygqjg4w54m2bshzevxy
846
TRUNC_HASH=bafykbzaceadeu7onzmlq7v33ytjpmo37rsqk2q6mzeqf5at55j32zxbcdbwig
847
test_files_api "($EXTRA, blake2b-256 root)"
@@ -866,10 +873,12 @@ test_expect_success "enable sharding in config" '
873
874
test_launch_ipfs_daemon_without_network
875
876
+# sharding cidv0: HAMT-sharded directory with 100 files, CIDv0
877
SHARD_HASH=QmPkwLJTYZRGPJ8Lazr9qPdrLmswPtUjaDbEpmR9jEh1se
878
test_sharding "(cidv0)"
879
872
-SHARD_HASH=bafybeib46tpawg2d2hhlmmn2jvgio33wqkhlehxrem7wbfvqqikure37rm
880
+# sharding cidv1: HAMT-sharded directory with 100 files, CIDv1
881
+SHARD_HASH=bafybeiaulcf7c46pqg3tkud6dsvbgvlnlhjuswcwtfhxts5c2kuvmh5keu
882
test_sharding "(cidv1 root)" "--cid-version=1"
883
884
test_kill_ipfs_daemon