fix: allow dag import of 1MiB chunks wrapped in dag-pb (#11185)
IPIP-499's unixfs-v1-2025 profile uses 1MiB chunks. with --raw-leaves=false, protobuf wrapping pushes blocks slightly over 1MiB. the previous 1MiB SoftBlockLimit rejected these blocks on dag import. raise SoftBlockLimit to 2MiB to match the bitswap spec, which requires implementations to support blocks up to 2MiB. - raise SoftBlockLimit to 2MiB per the bitswap spec - update error messages and help text - bump boxo to main with ipfs/boxo#1101 (raised ChunkSizeLimit/BlockSizeLimit, 256-byte overhead budget) - update sharness tests for 2MiB boundary - add test/cli boundary tests for block put, dag put, dag import, ipfs add (raw and wrapped leaves), and bitswap exchange including regression tests for the libp2p message size hard limit
Marcin Rataj committed
Feb 6, 2026 at 23:55 UTC
c6702eaf886b434fbe4452b438e277651e8a3b50
15 files changed
+466
-35
core/commands/add.go
+10
@@ -172,6 +172,16 @@ Buzhash or Rabin fingerprint chunker for content defined chunking by
172
specifying buzhash or rabin-[min]-[avg]-[max] (where min/avg/max refer
173
to the desired chunk sizes in bytes), e.g. 'rabin-262144-524288-1048576'.
174
175
+The maximum accepted value for 'size-N' and rabin 'max' parameter is
176
+2MiB minus 256 bytes (2096896 bytes). The 256-byte overhead budget is
177
+reserved for protobuf/UnixFS framing so that serialized blocks stay
178
+within the 2MiB block size limit from the bitswap spec. The buzhash
179
+chunker uses a fixed internal maximum of 512KiB and is not affected.
180
+
181
+Only the fixed-size chunker ('size-N') guarantees that the same data
182
+will always produce the same CID. The rabin and buzhash chunkers may
183
+change their internal parameters in a future release.
184
+
185
The following examples use very small byte sizes to demonstrate the
186
properties of the different chunkers on a small file. You'll likely
187
want to use a 1024 times larger chunk sizes for most files.
core/commands/cmdutils/utils.go
+8
-7
@@ -14,14 +14,16 @@ import (
14
15
const (
16
AllowBigBlockOptionName = "allow-big-block"
17
- SoftBlockLimit = 1024 * 1024 // https://github.com/ipfs/kubo/issues/7421#issuecomment-910833499
18
- MaxPinNameBytes = 255 // Maximum number of bytes allowed for a pin name
17
+ // SoftBlockLimit is the maximum block size for bitswap transfer.
18
+ // If this value changes, update the "2MiB" strings in error messages below.
19
+ SoftBlockLimit = 2 * 1024 * 1024 // https://specs.ipfs.tech/bitswap-protocol/#block-sizes
20
+ MaxPinNameBytes = 255 // Maximum number of bytes allowed for a pin name
21
)
22
23
var AllowBigBlockOption cmds.Option
24
25
func init() {
24
- AllowBigBlockOption = cmds.BoolOption(AllowBigBlockOptionName, "Disable block size check and allow creation of blocks bigger than 1MiB. WARNING: such blocks won't be transferable over the standard bitswap.").WithDefault(false)
26
+ AllowBigBlockOption = cmds.BoolOption(AllowBigBlockOptionName, "Disable block size check and allow creation of blocks bigger than 2MiB. WARNING: such blocks won't be transferable over the standard bitswap.").WithDefault(false)
27
}
28
29
func CheckCIDSize(req *cmds.Request, c cid.Cid, dagAPI coreiface.APIDagService) error {
@@ -44,11 +46,10 @@ func CheckBlockSize(req *cmds.Request, size uint64) error {
46
return nil
47
}
48
47
- // We do not allow producing blocks bigger than 1 MiB to avoid errors
48
- // when transmitting them over BitSwap. The 1 MiB constant is an
49
- // unenforced and undeclared rule of thumb hard-coded here.
49
+ // Block size is limited to SoftBlockLimit (2MiB) as defined in the bitswap spec.
50
+ // https://specs.ipfs.tech/bitswap-protocol/#block-sizes
51
if size > SoftBlockLimit {
51
- return fmt.Errorf("produced block is over 1MiB: big blocks can't be exchanged with other peers. consider using UnixFS for automatic chunking of bigger files, or pass --allow-big-block to override")
52
+ return fmt.Errorf("produced block is over 2MiB: big blocks can't be exchanged with other peers. consider using UnixFS for automatic chunking of bigger files, or pass --allow-big-block to override")
53
}
54
return nil
55
}
docs/changelogs/v0.40.md
+4
@@ -67,6 +67,10 @@ The `test-cid-v1` and `test-cid-v1-wide` profiles have been removed. Use `unixfs
67
68
When writing to MFS directories that use CIDv1 (via `--cid-version=1` or `ipfs files chcid`), single-block files now produce raw block CIDs (like `bafkrei...`), matching the behavior of `ipfs add --raw-leaves`. Previously, MFS would wrap single-block files in dag-pb even when raw leaves were enabled. CIDv0 directories continue to use dag-pb.
69
70
+**Block size limit raised to 2MiB**
71
+
72
+`ipfs block put`, `ipfs dag put`, and `ipfs dag import` now accept blocks up to 2MiB without `--allow-big-block`, matching the [bitswap spec](https://specs.ipfs.tech/bitswap-protocol/#block-sizes). The previous 1MiB limit was too restrictive and broke `ipfs dag import` of 1MiB-chunked non-raw-leaf data (protobuf wrapping pushes blocks slightly over 1MiB). The max `--chunker` value for `ipfs add` is `2MiB - 256 bytes` to leave room for protobuf framing. IPIP-499 profiles use lower chunk sizes (256KiB and 1MiB) and are not affected.
73
+
74
**HAMT Threshold Fix**
75
76
HAMT directory sharding threshold changed from `>=` to `>` to match the Go docs and JS implementation ([ipfs/boxo@6707376](https://github.com/ipfs/boxo/commit/6707376002a3d4ba64895749ce9be2e00d265ed5)). A directory exactly at 256 KiB now stays as a basic directory instead of converting to HAMT. This is a theoretical breaking change, but unlikely to impact real-world users as it requires a directory to be exactly at the threshold boundary. If you depend on the old behavior, adjust [`Import.UnixFSHAMTShardingSize`](https://github.com/ipfs/kubo/blob/master/docs/config.md#importunixfshamtshardingsize) to be 1 byte lower.
docs/config.md
+13
-1
@@ -3716,9 +3716,21 @@ The default UnixFS chunker. Commands affected: `ipfs add`.
3716
Valid formats:
3717
3718
- `size-<bytes>` - fixed size chunker
3719
-- `rabin-<min>-<avg>-<max>` - rabin fingerprint chunker
3719
+- `rabin-<min>-<avg>-<max>` - rabin fingerprint chunker
3720
- `buzhash` - buzhash chunker
3721
3722
+The maximum accepted value for `size-<bytes>` and rabin `max` parameter is
3723
+`2MiB - 256 bytes` (2096896 bytes). The 256-byte overhead budget is reserved
3724
+for protobuf/UnixFS framing so that serialized blocks stay within the 2MiB
3725
+block size limit defined by the
3726
+[bitswap spec](https://specs.ipfs.tech/bitswap-protocol/#block-sizes).
3727
+The `buzhash` chunker uses a fixed internal maximum of 512KiB and is not
3728
+affected by this limit.
3729
+
3730
+Only the fixed-size chunker (`size-<bytes>`) guarantees that the same data
3731
+will always produce the same CID. The `rabin` and `buzhash` chunkers may
3732
+change their internal parameters in a future release.
3733
+
3734
Default: `size-262144`
3735
3736
Type: `optionalString`
docs/examples/kubo-as-a-library/go.mod
+1
-1
@@ -7,7 +7,7 @@ go 1.25
7
replace github.com/ipfs/kubo => ./../../..
8
9
require (
10
- github.com/ipfs/boxo v0.36.1-0.20260205235512-2a942e3e1a75
10
+ github.com/ipfs/boxo v0.36.1-0.20260206224221-77bd614971f0
11
github.com/ipfs/kubo v0.0.0-00010101000000-000000000000
12
github.com/libp2p/go-libp2p v0.47.0
13
github.com/multiformats/go-multiaddr v0.16.1
docs/examples/kubo-as-a-library/go.sum
+2
-2
@@ -267,8 +267,8 @@ github.com/ipfs-shipyard/nopfs/ipfs v0.25.0 h1:OqNqsGZPX8zh3eFMO8Lf8EHRRnSGBMqcd
267
github.com/ipfs-shipyard/nopfs/ipfs v0.25.0/go.mod h1:BxhUdtBgOXg1B+gAPEplkg/GpyTZY+kCMSfsJvvydqU=
268
github.com/ipfs/bbloom v0.0.4 h1:Gi+8EGJ2y5qiD5FbsbpX/TMNcJw8gSqr7eyjHa4Fhvs=
269
github.com/ipfs/bbloom v0.0.4/go.mod h1:cS9YprKXpoZ9lT0n/Mw/a6/aFV6DTjTLYHeA+gyqMG0=
270
-github.com/ipfs/boxo v0.36.1-0.20260205235512-2a942e3e1a75 h1:1UoSAzXwwgOrCZm5cu6v6bL4OGYIzcaOew9Rl6ZycqQ=
271
-github.com/ipfs/boxo v0.36.1-0.20260205235512-2a942e3e1a75/go.mod h1:92hnRXfP5ScKEIqlq9Ns7LR1dFXEVADKWVGH0fjk83k=
270
+github.com/ipfs/boxo v0.36.1-0.20260206224221-77bd614971f0 h1:tC8iJdzsCy/npaez/gtQqNDLpl7DBqCARj9AECmYmoI=
271
+github.com/ipfs/boxo v0.36.1-0.20260206224221-77bd614971f0/go.mod h1:92hnRXfP5ScKEIqlq9Ns7LR1dFXEVADKWVGH0fjk83k=
272
github.com/ipfs/go-bitfield v1.1.0 h1:fh7FIo8bSwaJEh6DdTWbCeZ1eqOaOkKFI74SCnsWbGA=
273
github.com/ipfs/go-bitfield v1.1.0/go.mod h1:paqf1wjq/D2BBmzfTVFlJQ9IlFOZpg422HL0HqsGWHU=
274
github.com/ipfs/go-block-format v0.0.3/go.mod h1:4LmD4ZUw0mhO+JSKdpWwrzATiEfM7WWgQ8H5l6P8MVk=
go.mod
+2
-1
@@ -21,7 +21,7 @@ require (
21
github.com/hashicorp/go-version v1.8.0
22
github.com/ipfs-shipyard/nopfs v0.0.14
23
github.com/ipfs-shipyard/nopfs/ipfs v0.25.0
24
- github.com/ipfs/boxo v0.36.1-0.20260205235512-2a942e3e1a75
24
+ github.com/ipfs/boxo v0.36.1-0.20260206224221-77bd614971f0
25
github.com/ipfs/go-block-format v0.2.3
26
github.com/ipfs/go-cid v0.6.0
27
github.com/ipfs/go-cidutil v0.1.0
@@ -274,6 +274,7 @@ require (
274
)
275
276
// Exclude ancient +incompatible versions that confuse Dependabot.
277
+
278
// These pre-Go-modules versions reference packages that no longer exist.
279
exclude (
280
github.com/ipfs/go-ipfs-cmds v2.0.1+incompatible
go.sum
+2
-2
@@ -337,8 +337,8 @@ github.com/ipfs-shipyard/nopfs/ipfs v0.25.0 h1:OqNqsGZPX8zh3eFMO8Lf8EHRRnSGBMqcd
337
github.com/ipfs-shipyard/nopfs/ipfs v0.25.0/go.mod h1:BxhUdtBgOXg1B+gAPEplkg/GpyTZY+kCMSfsJvvydqU=
338
github.com/ipfs/bbloom v0.0.4 h1:Gi+8EGJ2y5qiD5FbsbpX/TMNcJw8gSqr7eyjHa4Fhvs=
339
github.com/ipfs/bbloom v0.0.4/go.mod h1:cS9YprKXpoZ9lT0n/Mw/a6/aFV6DTjTLYHeA+gyqMG0=
340
-github.com/ipfs/boxo v0.36.1-0.20260205235512-2a942e3e1a75 h1:1UoSAzXwwgOrCZm5cu6v6bL4OGYIzcaOew9Rl6ZycqQ=
341
-github.com/ipfs/boxo v0.36.1-0.20260205235512-2a942e3e1a75/go.mod h1:92hnRXfP5ScKEIqlq9Ns7LR1dFXEVADKWVGH0fjk83k=
340
+github.com/ipfs/boxo v0.36.1-0.20260206224221-77bd614971f0 h1:tC8iJdzsCy/npaez/gtQqNDLpl7DBqCARj9AECmYmoI=
341
+github.com/ipfs/boxo v0.36.1-0.20260206224221-77bd614971f0/go.mod h1:92hnRXfP5ScKEIqlq9Ns7LR1dFXEVADKWVGH0fjk83k=
342
github.com/ipfs/go-bitfield v1.1.0 h1:fh7FIo8bSwaJEh6DdTWbCeZ1eqOaOkKFI74SCnsWbGA=
343
github.com/ipfs/go-bitfield v1.1.0/go.mod h1:paqf1wjq/D2BBmzfTVFlJQ9IlFOZpg422HL0HqsGWHU=
344
github.com/ipfs/go-block-format v0.0.3/go.mod h1:4LmD4ZUw0mhO+JSKdpWwrzATiEfM7WWgQ8H5l6P8MVk=
test/cli/block_size_test.go
new
+403
@@ -0,0 +1,403 @@
1
+package cli
2
+
3
+import (
4
+ "bytes"
5
+ "crypto/rand"
6
+ "encoding/json"
7
+ "fmt"
8
+ "os"
9
+ "path/filepath"
10
+ "strings"
11
+ "testing"
12
+ "time"
13
+
14
+ "github.com/ipfs/kubo/test/cli/harness"
15
+ "github.com/stretchr/testify/assert"
16
+ "github.com/stretchr/testify/require"
17
+)
18
+
19
+const (
20
+ twoMiB = 2 * 1024 * 1024 // 2097152 - bitswap spec block size limit
21
+ twoMiBPlus = twoMiB + 1 // 2097153
22
+ maxChunkSize = twoMiB - 256 // 2096896 - max chunker value (overhead budget for protobuf framing)
23
+ overMaxChunk = maxChunkSize + 1 // 2096897
24
+
25
+ // go-libp2p v0.47.0 network.MessageSizeMax is 4194304 bytes (4MiB).
26
+ // A bitswap message carrying a single block has a protobuf envelope
27
+ // whose size depends on the CID used to represent the block. For
28
+ // CIDv1 with raw codec and SHA2-256 multihash (4-byte CID prefix),
29
+ // the envelope is 18 bytes: 2 bytes for the empty Wantlist submessage,
30
+ // 6 bytes for the CID prefix field, 5 bytes for field tags and the
31
+ // payload length varint, and 5 bytes for the data length varint and
32
+ // block submessage length varint. The msgio varint reader rejects
33
+ // messages strictly larger than MessageSizeMax, so the maximum block
34
+ // that fits is 4194304 - 18 = 4194286 bytes.
35
+ //
36
+ // The hard limit varies slightly depending on the CID: a longer
37
+ // multihash (e.g. SHA-512) increases the CID prefix and reduces the
38
+ // maximum block payload by the same amount.
39
+ libp2pMsgMax = 4 * 1024 * 1024 // 4194304 - libp2p network.MessageSizeMax
40
+ bsBlockEnvelope = 18 // protobuf overhead for CIDv1 + raw + SHA2-256
41
+ maxTransferBlock = libp2pMsgMax - bsBlockEnvelope // 4194286 - largest block transferable via bitswap
42
+ overMaxTransfer = maxTransferBlock + 1 // 4194287
43
+)
44
+
45
+// blockSize returns the block size in bytes for a given CID by parsing
46
+// the JSON output of `ipfs block stat --enc=json <cid>`.
47
+func blockSize(t *testing.T, node *harness.Node, cid string) int {
48
+ t.Helper()
49
+ res := node.IPFS("block", "stat", "--enc=json", cid)
50
+ var stat struct {
51
+ Key string
52
+ Size int
53
+ }
54
+ require.NoError(t, json.Unmarshal(res.Stdout.Bytes(), &stat))
55
+ return stat.Size
56
+}
57
+
58
+// allBlockCIDs returns the root CID plus all recursive refs for a DAG.
59
+func allBlockCIDs(t *testing.T, node *harness.Node, root string) []string {
60
+ t.Helper()
61
+ cids := []string{root}
62
+ res := node.IPFS("refs", "-r", "--unique", root)
63
+ for _, line := range strings.Split(strings.TrimSpace(res.Stdout.String()), "\n") {
64
+ if line != "" {
65
+ cids = append(cids, line)
66
+ }
67
+ }
68
+ return cids
69
+}
70
+
71
+// assertAllBlocksWithinLimit checks that every block in the DAG rooted at
72
+// root is at most twoMiB bytes.
73
+func assertAllBlocksWithinLimit(t *testing.T, node *harness.Node, root string) {
74
+ t.Helper()
75
+ for _, c := range allBlockCIDs(t, node, root) {
76
+ size := blockSize(t, node, c)
77
+ assert.LessOrEqual(t, size, twoMiB, fmt.Sprintf("block %s is %d bytes, exceeds 2MiB limit", c, size))
78
+ }
79
+}
80
+
81
+func TestBlockSizeBoundary(t *testing.T) {
82
+ t.Parallel()
83
+
84
+ t.Run("block put", func(t *testing.T) {
85
+ t.Parallel()
86
+
87
+ t.Run("exactly 2MiB succeeds", func(t *testing.T) {
88
+ t.Parallel()
89
+ node := harness.NewT(t).NewNode().Init().StartDaemon("--offline")
90
+ defer node.StopDaemon()
91
+
92
+ data := make([]byte, twoMiB)
93
+ cid := strings.TrimSpace(
94
+ node.PipeToIPFS(bytes.NewReader(data), "block", "put").Stdout.String(),
95
+ )
96
+ got := node.IPFS("block", "get", cid)
97
+ assert.Len(t, got.Stdout.Bytes(), twoMiB)
98
+ })
99
+
100
+ t.Run("2MiB+1 fails without --allow-big-block", func(t *testing.T) {
101
+ t.Parallel()
102
+ node := harness.NewT(t).NewNode().Init().StartDaemon("--offline")
103
+ defer node.StopDaemon()
104
+
105
+ data := make([]byte, twoMiBPlus)
106
+ res := node.RunPipeToIPFS(bytes.NewReader(data), "block", "put")
107
+ assert.NotEqual(t, 0, res.ExitCode())
108
+ assert.Contains(t, res.Stderr.String(), "produced block is over 2MiB: big blocks can't be exchanged with other peers. consider using UnixFS for automatic chunking of bigger files, or pass --allow-big-block to override")
109
+ })
110
+
111
+ t.Run("2MiB+1 succeeds with --allow-big-block", func(t *testing.T) {
112
+ t.Parallel()
113
+ node := harness.NewT(t).NewNode().Init().StartDaemon("--offline")
114
+ defer node.StopDaemon()
115
+
116
+ data := make([]byte, twoMiBPlus)
117
+ cid := strings.TrimSpace(
118
+ node.PipeToIPFS(bytes.NewReader(data), "block", "put", "--allow-big-block").Stdout.String(),
119
+ )
120
+ got := node.IPFS("block", "get", cid)
121
+ assert.Len(t, got.Stdout.Bytes(), twoMiBPlus)
122
+ })
123
+ })
124
+
125
+ t.Run("dag put", func(t *testing.T) {
126
+ t.Parallel()
127
+
128
+ t.Run("exactly 2MiB succeeds", func(t *testing.T) {
129
+ t.Parallel()
130
+ node := harness.NewT(t).NewNode().Init().StartDaemon("--offline")
131
+ defer node.StopDaemon()
132
+
133
+ data := make([]byte, twoMiB)
134
+ cid := strings.TrimSpace(
135
+ node.PipeToIPFS(bytes.NewReader(data), "dag", "put", "--input-codec=raw", "--store-codec=raw").Stdout.String(),
136
+ )
137
+ got := node.IPFS("block", "get", cid)
138
+ assert.Len(t, got.Stdout.Bytes(), twoMiB)
139
+ })
140
+
141
+ t.Run("2MiB+1 fails without --allow-big-block", func(t *testing.T) {
142
+ t.Parallel()
143
+ node := harness.NewT(t).NewNode().Init().StartDaemon("--offline")
144
+ defer node.StopDaemon()
145
+
146
+ data := make([]byte, twoMiBPlus)
147
+ res := node.RunPipeToIPFS(bytes.NewReader(data), "dag", "put", "--input-codec=raw", "--store-codec=raw")
148
+ assert.NotEqual(t, 0, res.ExitCode())
149
+ assert.Contains(t, res.Stderr.String(), "produced block is over 2MiB: big blocks can't be exchanged with other peers. consider using UnixFS for automatic chunking of bigger files, or pass --allow-big-block to override")
150
+ })
151
+
152
+ t.Run("2MiB+1 succeeds with --allow-big-block", func(t *testing.T) {
153
+ t.Parallel()
154
+ node := harness.NewT(t).NewNode().Init().StartDaemon("--offline")
155
+ defer node.StopDaemon()
156
+
157
+ data := make([]byte, twoMiBPlus)
158
+ cid := strings.TrimSpace(
159
+ node.PipeToIPFS(bytes.NewReader(data), "dag", "put", "--input-codec=raw", "--store-codec=raw", "--allow-big-block").Stdout.String(),
160
+ )
161
+ got := node.IPFS("block", "get", cid)
162
+ assert.Len(t, got.Stdout.Bytes(), twoMiBPlus)
163
+ })
164
+ })
165
+
166
+ t.Run("dag import and export", func(t *testing.T) {
167
+ t.Parallel()
168
+
169
+ t.Run("2MiB+1 block round-trips with --allow-big-block", func(t *testing.T) {
170
+ t.Parallel()
171
+ node := harness.NewT(t).NewNode().Init().StartDaemon("--offline")
172
+ defer node.StopDaemon()
173
+
174
+ // put an oversized raw block with override
175
+ data := make([]byte, twoMiBPlus)
176
+ cid := strings.TrimSpace(
177
+ node.PipeToIPFS(bytes.NewReader(data), "dag", "put", "--input-codec=raw", "--store-codec=raw", "--allow-big-block").Stdout.String(),
178
+ )
179
+
180
+ // export to CAR
181
+ carPath := filepath.Join(node.Dir, "oversized.car")
182
+ require.NoError(t, node.IPFSDagExport(cid, carPath))
183
+
184
+ // re-import without --allow-big-block should fail
185
+ carFile, err := os.Open(carPath)
186
+ require.NoError(t, err)
187
+ res := node.RunPipeToIPFS(carFile, "dag", "import")
188
+ carFile.Close()
189
+ assert.NotEqual(t, 0, res.ExitCode())
190
+ assert.Contains(t, res.Stderr.String()+res.Stdout.String(), "produced block is over 2MiB: big blocks can't be exchanged with other peers. consider using UnixFS for automatic chunking of bigger files, or pass --allow-big-block to override")
191
+
192
+ // re-import with --allow-big-block should succeed
193
+ carFile, err = os.Open(carPath)
194
+ require.NoError(t, err)
195
+ res = node.RunPipeToIPFS(carFile, "dag", "import", "--allow-big-block")
196
+ carFile.Close()
197
+ assert.Equal(t, 0, res.ExitCode())
198
+ })
199
+ })
200
+
201
+ t.Run("ipfs add non-raw-leaves", func(t *testing.T) {
202
+ t.Parallel()
203
+
204
+ // The chunker enforces ChunkSizeLimit (maxChunkSize = 2MiB - 256
205
+ // as of boxo 2026Q1) regardless of leaf type. It does not know at parse time whether
206
+ // raw or wrapped leaves will be used, so the 256-byte overhead
207
+ // budget is applied uniformly.
208
+ //
209
+ // With --raw-leaves=false each chunk is wrapped in protobuf,
210
+ // adding ~14 bytes overhead that pushes blocks past the chunk size.
211
+ // The overhead budget ensures the wrapped block stays within 2MiB.
212
+ //
213
+ // With --raw-leaves=true there is no protobuf wrapper, so the
214
+ // block is exactly the chunk size (maxChunkSize). The 256-byte
215
+ // budget is unused in this case but the chunker still enforces it.
216
+ // A full 2MiB chunk (--chunker=size-2097152) is rejected even
217
+ // though the resulting raw block would fit within BlockSizeLimit.
218
+
219
+ t.Run("1MiB chunk with protobuf wrapping succeeds under 2MiB limit", func(t *testing.T) {
220
+ t.Parallel()
221
+ node := harness.NewT(t).NewNode().Init().StartDaemon("--offline")
222
+ defer node.StopDaemon()
223
+
224
+ data := make([]byte, twoMiB)
225
+ res := node.RunPipeToIPFS(bytes.NewReader(data), "add", "-q", "--chunker=size-1048576", "--raw-leaves=false")
226
+ require.Equal(t, 0, res.ExitCode(), "stderr: %s", res.Stderr.String())
227
+ root := strings.TrimSpace(res.Stdout.String())
228
+ // the last line of `ipfs add -q` is the root CID
229
+ lines := strings.Split(root, "\n")
230
+ root = lines[len(lines)-1]
231
+ assertAllBlocksWithinLimit(t, node, root)
232
+ })
233
+
234
+ t.Run("max chunk with protobuf wrapping stays within block limit", func(t *testing.T) {
235
+ t.Parallel()
236
+ node := harness.NewT(t).NewNode().Init().StartDaemon("--offline")
237
+ defer node.StopDaemon()
238
+
239
+ // maxChunkSize leaves room for protobuf framing overhead
240
+ data := make([]byte, maxChunkSize*2)
241
+ res := node.RunPipeToIPFS(bytes.NewReader(data), "add", "-q",
242
+ fmt.Sprintf("--chunker=size-%d", maxChunkSize), "--raw-leaves=false")
243
+ require.Equal(t, 0, res.ExitCode(), "stderr: %s", res.Stderr.String())
244
+ lines := strings.Split(strings.TrimSpace(res.Stdout.String()), "\n")
245
+ root := lines[len(lines)-1]
246
+ assertAllBlocksWithinLimit(t, node, root)
247
+ })
248
+
249
+ t.Run("chunk size over limit is rejected by chunker", func(t *testing.T) {
250
+ t.Parallel()
251
+ node := harness.NewT(t).NewNode().Init().StartDaemon("--offline")
252
+ defer node.StopDaemon()
253
+
254
+ data := make([]byte, twoMiB+twoMiB)
255
+ res := node.RunPipeToIPFS(bytes.NewReader(data), "add", "-q",
256
+ fmt.Sprintf("--chunker=size-%d", overMaxChunk), "--raw-leaves=false")
257
+ assert.NotEqual(t, 0, res.ExitCode())
258
+ assert.Contains(t, res.Stderr.String(),
259
+ fmt.Sprintf("chunker parameters may not exceed the maximum chunk size of %d", maxChunkSize))
260
+ })
261
+
262
+ t.Run("max chunk with raw leaves succeeds", func(t *testing.T) {
263
+ t.Parallel()
264
+ node := harness.NewT(t).NewNode().Init().StartDaemon("--offline")
265
+ defer node.StopDaemon()
266
+
267
+ // raw leaves have no protobuf wrapper, so max chunk size fits easily
268
+ data := make([]byte, maxChunkSize*2)
269
+ res := node.RunPipeToIPFS(bytes.NewReader(data), "add", "-q",
270
+ fmt.Sprintf("--chunker=size-%d", maxChunkSize), "--raw-leaves=true")
271
+ require.Equal(t, 0, res.ExitCode(), "stderr: %s", res.Stderr.String())
272
+ lines := strings.Split(strings.TrimSpace(res.Stdout.String()), "\n")
273
+ root := lines[len(lines)-1]
274
+ assertAllBlocksWithinLimit(t, node, root)
275
+ })
276
+ })
277
+
278
+ t.Run("bitswap exchange", func(t *testing.T) {
279
+ t.Parallel()
280
+
281
+ t.Run("2MiB raw block transfers between peers", func(t *testing.T) {
282
+ t.Parallel()
283
+ h := harness.NewT(t)
284
+ provider := h.NewNode().Init("--profile=unixfs-v1-2025").StartDaemon()
285
+ defer provider.StopDaemon()
286
+ requester := h.NewNode().Init("--profile=unixfs-v1-2025").StartDaemon()
287
+ defer requester.StopDaemon()
288
+
289
+ data := make([]byte, twoMiB)
290
+ _, err := rand.Read(data)
291
+ require.NoError(t, err)
292
+ cid := strings.TrimSpace(
293
+ provider.PipeToIPFS(bytes.NewReader(data), "block", "put").Stdout.String(),
294
+ )
295
+
296
+ requester.Connect(provider)
297
+
298
+ res := requester.IPFS("block", "get", cid)
299
+ assert.Equal(t, data, res.Stdout.Bytes(), "retrieved block should match original")
300
+ })
301
+
302
+ t.Run("unixfs-v1-2025: 2MiB file transfers between peers", func(t *testing.T) {
303
+ t.Parallel()
304
+ h := harness.NewT(t)
305
+ provider := h.NewNode().Init("--profile=unixfs-v1-2025").StartDaemon()
306
+ defer provider.StopDaemon()
307
+ requester := h.NewNode().Init("--profile=unixfs-v1-2025").StartDaemon()
308
+ defer requester.StopDaemon()
309
+
310
+ // unixfs-v1-2025 profile uses CIDv1, raw leaves, SHA2-256,
311
+ // and 1MiB chunks. A 2MiB file produces two 1MiB raw leaf
312
+ // blocks plus a root node, all within the 2MiB spec limit.
313
+ data := make([]byte, twoMiB)
314
+ _, err := rand.Read(data)
315
+ require.NoError(t, err)
316
+ res := provider.RunPipeToIPFS(bytes.NewReader(data), "add", "-q")
317
+ require.Equal(t, 0, res.ExitCode(), "stderr: %s", res.Stderr.String())
318
+ lines := strings.Split(strings.TrimSpace(res.Stdout.String()), "\n")
319
+ root := lines[len(lines)-1]
320
+
321
+ requester.Connect(provider)
322
+
323
+ got := requester.IPFS("cat", root)
324
+ assert.Equal(t, data, got.Stdout.Bytes(), "retrieved file should match original")
325
+ })
326
+
327
+ // The following two tests guard the physical hard limit of the
328
+ // libp2p transport layer (network.MessageSizeMax = 4MiB). This is
329
+ // the actual ceiling for bitswap block transfer, independent of the
330
+ // 2MiB soft limit from the bitswap spec. Knowing the exact hard
331
+ // limit is important for backward-compatible protocol and standards
332
+ // evolution: any future increase to the bitswap spec block size
333
+ // must stay within the libp2p message framing budget, or the
334
+ // transport layer must be updated first.
335
+
336
+ t.Run("bitswap-over-libp2p: largest block that fits in message transfers", func(t *testing.T) {
337
+ t.Parallel()
338
+ h := harness.NewT(t)
339
+ provider := h.NewNode().Init("--profile=unixfs-v1-2025").StartDaemon()
340
+ defer provider.StopDaemon()
341
+ requester := h.NewNode().Init("--profile=unixfs-v1-2025").StartDaemon()
342
+ defer requester.StopDaemon()
343
+
344
+ data := make([]byte, maxTransferBlock)
345
+ _, err := rand.Read(data)
346
+ require.NoError(t, err)
347
+ cid := strings.TrimSpace(
348
+ provider.PipeToIPFS(bytes.NewReader(data), "block", "put", "--allow-big-block").Stdout.String(),
349
+ )
350
+
351
+ requester.Connect(provider)
352
+
353
+ // successful transfers complete in ~1s
354
+ timeout := time.After(5 * time.Second)
355
+ dataChan := make(chan []byte, 1)
356
+
357
+ go func() {
358
+ res := requester.RunIPFS("block", "get", cid)
359
+ dataChan <- res.Stdout.Bytes()
360
+ }()
361
+
362
+ select {
363
+ case got := <-dataChan:
364
+ assert.Equal(t, data, got, "retrieved block should match original")
365
+ case <-timeout:
366
+ t.Fatal("block get timed out: expected transfer to succeed at maxTransferBlock")
367
+ }
368
+ })
369
+
370
+ t.Run("bitswap-over-libp2p: one byte over message limit does not transfer", func(t *testing.T) {
371
+ t.Parallel()
372
+ h := harness.NewT(t)
373
+ provider := h.NewNode().Init("--profile=unixfs-v1-2025").StartDaemon()
374
+ defer provider.StopDaemon()
375
+ requester := h.NewNode().Init("--profile=unixfs-v1-2025").StartDaemon()
376
+ defer requester.StopDaemon()
377
+
378
+ data := make([]byte, overMaxTransfer)
379
+ _, err := rand.Read(data)
380
+ require.NoError(t, err)
381
+ cid := strings.TrimSpace(
382
+ provider.PipeToIPFS(bytes.NewReader(data), "block", "put", "--allow-big-block").Stdout.String(),
383
+ )
384
+
385
+ requester.Connect(provider)
386
+
387
+ timeout := time.After(5 * time.Second)
388
+ dataChan := make(chan []byte, 1)
389
+
390
+ go func() {
391
+ res := requester.RunIPFS("block", "get", cid)
392
+ dataChan <- res.Stdout.Bytes()
393
+ }()
394
+
395
+ select {
396
+ case got := <-dataChan:
397
+ t.Fatalf("expected timeout, but block was retrieved (%d bytes)", len(got))
398
+ case <-timeout:
399
+ t.Log("block get timed out as expected: block exceeds libp2p message size limit")
400
+ }
401
+ })
402
+ })
403
+}
test/dependencies/go.mod
+1
-1
@@ -135,7 +135,7 @@ require (
135
github.com/huin/goupnp v1.3.0 // indirect
136
github.com/inconshreveable/mousetrap v1.1.0 // indirect
137
github.com/ipfs/bbloom v0.0.4 // indirect
138
- github.com/ipfs/boxo v0.36.1-0.20260205235512-2a942e3e1a75 // indirect
138
+ github.com/ipfs/boxo v0.36.1-0.20260206224221-77bd614971f0 // indirect
139
github.com/ipfs/go-bitfield v1.1.0 // indirect
140
github.com/ipfs/go-block-format v0.2.3 // indirect
141
github.com/ipfs/go-cid v0.6.0 // indirect
test/dependencies/go.sum
+2
-2
@@ -296,8 +296,8 @@ github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2
296
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
297
github.com/ipfs/bbloom v0.0.4 h1:Gi+8EGJ2y5qiD5FbsbpX/TMNcJw8gSqr7eyjHa4Fhvs=
298
github.com/ipfs/bbloom v0.0.4/go.mod h1:cS9YprKXpoZ9lT0n/Mw/a6/aFV6DTjTLYHeA+gyqMG0=
299
-github.com/ipfs/boxo v0.36.1-0.20260205235512-2a942e3e1a75 h1:1UoSAzXwwgOrCZm5cu6v6bL4OGYIzcaOew9Rl6ZycqQ=
300
-github.com/ipfs/boxo v0.36.1-0.20260205235512-2a942e3e1a75/go.mod h1:92hnRXfP5ScKEIqlq9Ns7LR1dFXEVADKWVGH0fjk83k=
299
+github.com/ipfs/boxo v0.36.1-0.20260206224221-77bd614971f0 h1:tC8iJdzsCy/npaez/gtQqNDLpl7DBqCARj9AECmYmoI=
300
+github.com/ipfs/boxo v0.36.1-0.20260206224221-77bd614971f0/go.mod h1:92hnRXfP5ScKEIqlq9Ns7LR1dFXEVADKWVGH0fjk83k=
301
github.com/ipfs/go-bitfield v1.1.0 h1:fh7FIo8bSwaJEh6DdTWbCeZ1eqOaOkKFI74SCnsWbGA=
302
github.com/ipfs/go-bitfield v1.1.0/go.mod h1:paqf1wjq/D2BBmzfTVFlJQ9IlFOZpg422HL0HqsGWHU=
303
github.com/ipfs/go-block-format v0.2.3 h1:mpCuDaNXJ4wrBJLrtEaGFGXkferrw5eqVvzaHhtFKQk=
test/sharness/t0050-block.sh
+5
-5
@@ -291,17 +291,17 @@ test_expect_success "put with sha3 and cidv0 fails" '
291
'
292
293
test_expect_success "'ipfs block put' check block size" '
294
- dd if=/dev/zero bs=2MB count=1 > 2-MB-file &&
295
- test_expect_code 1 ipfs block put 2-MB-file >block_put_out 2>&1
294
+ dd if=/dev/zero bs=2097153 count=1 > over-2MiB-file &&
295
+ test_expect_code 1 ipfs block put over-2MiB-file >block_put_out 2>&1
296
'
297
298
test_expect_success "ipfs block put output has the correct error" '
299
- grep "produced block is over 1MiB" block_put_out
299
+ grep "produced block is over 2MiB" block_put_out
300
'
301
302
test_expect_success "ipfs block put --allow-big-block=true works" '
303
- test_expect_code 0 ipfs block put 2-MB-file --allow-big-block=true &&
304
- rm 2-MB-file
303
+ test_expect_code 0 ipfs block put over-2MiB-file --allow-big-block=true &&
304
+ rm over-2MiB-file
305
'
306
307
test_done
test/sharness/t0051-object.sh
+3
-3
@@ -42,16 +42,16 @@ test_object_cmd() {
42
43
test_expect_success "'ipfs object patch' check output block size" '
44
DIR=$EMPTY_UNIXFS_DIR
45
- for i in {1..13}
45
+ for i in {1..14}
46
do
47
DIR=$(ipfs object patch "$DIR" add-link "$DIR.jpg" "$DIR")
48
done
49
- # Fail when new block goes over the BS limit of 1MiB, but allow manual override
49
+ # Fail when new block goes over the BS limit of 2MiB, but allow manual override
50
test_expect_code 1 ipfs object patch "$DIR" add-link "$DIR.jpg" "$DIR" >patch_out 2>&1
51
'
52
53
test_expect_success "ipfs object patch add-link output has the correct error" '
54
- grep "produced block is over 1MiB" patch_out
54
+ grep "produced block is over 2MiB" patch_out
55
'
56
57
test_expect_success "ipfs object patch --allow-big-block=true add-link works" '
test/sharness/t0053-dag.sh
+5
-5
@@ -45,17 +45,17 @@ test_dag_cmd() {
45
'
46
47
test_expect_success "'ipfs dag put' check block size" '
48
- dd if=/dev/zero bs=2MB count=1 > 2-MB-file &&
49
- test_expect_code 1 ipfs dag put --input-codec=raw --store-codec=raw 2-MB-file >dag_put_out 2>&1
48
+ dd if=/dev/zero bs=2097153 count=1 > over-2MiB-file &&
49
+ test_expect_code 1 ipfs dag put --input-codec=raw --store-codec=raw over-2MiB-file >dag_put_out 2>&1
50
'
51
52
test_expect_success "ipfs dag put output has the correct error" '
53
- grep "produced block is over 1MiB" dag_put_out
53
+ grep "produced block is over 2MiB" dag_put_out
54
'
55
56
test_expect_success "ipfs dag put --allow-big-block=true works" '
57
- test_expect_code 0 ipfs dag put --input-codec=raw --store-codec=raw 2-MB-file --allow-big-block=true &&
58
- rm 2-MB-file
57
+ test_expect_code 0 ipfs dag put --input-codec=raw --store-codec=raw over-2MiB-file --allow-big-block=true &&
58
+ rm over-2MiB-file
59
'
60
61
test_expect_success "can add an ipld object using dag-json to dag-json" '
test/sharness/t0054-dag-car-import-export.sh
+5
-5
@@ -232,16 +232,16 @@ test_expect_success "naked root import expected output" '
232
'
233
234
test_expect_success "'ipfs dag import' check block size" '
235
- BIG_CID=$(dd if=/dev/zero bs=2MB count=1 | ipfs dag put --input-codec=raw --store-codec=raw --allow-big-block) &&
236
- ipfs dag export $BIG_CID > 2-MB-block.car &&
237
- test_expect_code 1 ipfs dag import 2-MB-block.car >dag_import_out 2>&1
235
+ BIG_CID=$(dd if=/dev/zero bs=2097153 count=1 | ipfs dag put --input-codec=raw --store-codec=raw --allow-big-block) &&
236
+ ipfs dag export $BIG_CID > over-2MiB-block.car &&
237
+ test_expect_code 1 ipfs dag import over-2MiB-block.car >dag_import_out 2>&1
238
'
239
test_expect_success "ipfs dag import output has the correct error" '
240
- grep "block is over 1MiB" dag_import_out
240
+ grep "block is over 2MiB" dag_import_out
241
'
242
243
test_expect_success "ipfs dag import --allow-big-block works" '
244
- test_expect_code 0 ipfs dag import --allow-big-block 2-MB-block.car
244
+ test_expect_code 0 ipfs dag import --allow-big-block over-2MiB-block.car
245
'
246
247
cat > version_2_import_expected << EOE