@cryptotaxi247 / kubo
Code Commits Issues Pulls Sessions CI/CD
master
master
md 209 lines 7.46 KB
Rendered Raw
1 # How `ipfs add` Works
2
3 This document explains what happens when you run `ipfs add` to import files into IPFS. Understanding this flow helps when debugging, optimizing imports, or building applications on top of IPFS.
4
5 - [The Big Picture](#the-big-picture)
6 - [Try It Yourself](#try-it-yourself)
7 - [Step by Step](#step-by-step)
8 - [Step 1: Chunking](#step-1-chunking)
9 - [Step 2: Building the DAG](#step-2-building-the-dag)
10 - [Step 3: Storing Blocks](#step-3-storing-blocks)
11 - [Step 4: Pinning](#step-4-pinning)
12 - [Alternative: Organizing with MFS](#alternative-organizing-with-mfs)
13 - [Options](#options)
14 - [UnixFS Format](#unixfs-format)
15 - [Code Architecture](#code-architecture)
16 - [Key Files](#key-files)
17 - [The Adder](#the-adder)
18 - [Further Reading](#further-reading)
19
20 ## The Big Picture
21
22 When you add a file to IPFS, three main things happen:
23
24 1. **Chunking** - The file is split into smaller pieces
25 2. **DAG Building** - Those pieces are organized into a tree structure (a [Merkle DAG](https://docs.ipfs.tech/concepts/merkle-dag/))
26 3. **Pinning** - The root of the tree is pinned so it persists in your local node
27
28 The result is a Content Identifier (CID) - a hash that uniquely identifies your content and can be used to retrieve it from anywhere in the IPFS network.
29
30 ```mermaid
31 flowchart LR
32 A["Your File<br/>(bytes)"] --> B["Chunker<br/>(split data)"]
33 B --> C["DAG Builder<br/>(tree)"]
34 C --> D["CID<br/>(hash)"]
35 ```
36
37 ## Try It Yourself
38
39 ```bash
40 # Add a simple file
41 echo "Hello World" > hello.txt
42 ipfs add hello.txt
43 # added QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy47CnJDgvs8u hello.txt
44
45 # See what's inside
46 ipfs cat QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy47CnJDgvs8u
47 # Hello World
48
49 # View the DAG structure
50 ipfs dag get QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy47CnJDgvs8u
51 ```
52
53 ## Step by Step
54
55 ### Step 1: Chunking
56
57 Big files are split into chunks because:
58
59 - Large files need to be broken down for efficient transfer
60 - Identical chunks across files are stored only once (deduplication)
61 - You can fetch parts of a file without downloading the whole thing
62
63 **Chunking strategies** (set with `--chunker`):
64
65 | Strategy | Description | Best For |
66 |----------|-------------|----------|
67 | `size-N` | Fixed size chunks | General use |
68 | `rabin` | Content-defined chunks using rolling hash | Deduplication across similar files |
69 | `buzhash` | Alternative content-defined chunking | Similar to rabin |
70
71 See `ipfs add --help` for current defaults, or [Import](config.md#import) for making them permanent.
72
73 Content-defined chunking (rabin/buzhash) finds natural boundaries in the data. This means if you edit the middle of a file, only the changed chunks need to be re-stored - the rest can be deduplicated.
74
75 ### Step 2: Building the DAG
76
77 Each chunk becomes a leaf node in a tree. If a file has many chunks, intermediate nodes group them together. This creates a Merkle DAG (Directed Acyclic Graph) where:
78
79 - Each node is identified by a hash of its contents
80 - Parent nodes contain links (hashes) to their children
81 - The root node's hash becomes the file's CID
82
83 **Layout strategies**:
84
85 **Balanced layout** (default):
86
87 ```mermaid
88 graph TD
89 Root --> Node1[Node]
90 Root --> Node2[Node]
91 Node1 --> Leaf1[Leaf]
92 Node1 --> Leaf2[Leaf]
93 Node2 --> Leaf3[Leaf]
94 ```
95
96 All leaves at similar depth. Good for random access - you can jump to any part of the file efficiently.
97
98 **Trickle layout** (`--trickle`):
99
100 ```mermaid
101 graph TD
102 Root --> Leaf1[Leaf]
103 Root --> Node1[Node]
104 Root --> Node2[Node]
105 Node1 --> Leaf2[Leaf]
106 Node2 --> Leaf3[Leaf]
107 ```
108
109 Leaves added progressively. Good for streaming - you can start reading before the whole file is added.
110
111 ### Step 3: Storing Blocks
112
113 As the DAG is built, each node is stored in the blockstore:
114
115 - **Normal mode**: Data is copied into IPFS's internal storage (`~/.ipfs/blocks/`)
116 - **Filestore mode** (`--nocopy`): Only references to the original file are stored (saves disk space but the original file must remain in place)
117
118 ### Step 4: Pinning
119
120 By default, added content is pinned (`ipfs add --pin=true`). This tells your IPFS node to keep this data - without pinning, content may eventually be removed to free up space.
121
122 ### Alternative: Organizing with MFS
123
124 Instead of pinning, you can use the [Mutable File System (MFS)](https://docs.ipfs.tech/concepts/file-systems/#mutable-file-system-mfs) to organize content using familiar paths like `/photos/vacation.jpg` instead of raw CIDs:
125
126 ```bash
127 # Add directly to MFS path
128 ipfs add --to-files=/backups/ myfile.txt
129
130 # Or copy an existing CID into MFS
131 ipfs files cp /ipfs/QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy47CnJDgvs8u /docs/hello.txt
132 ```
133
134 Content in MFS is implicitly pinned and stays organized across node restarts.
135
136 ## Options
137
138 Run `ipfs add --help` to see all available options for controlling chunking, DAG layout, CID format, pinning behavior, and more.
139
140 ## UnixFS Format
141
142 IPFS uses [UnixFS](https://specs.ipfs.tech/unixfs/) to represent files and directories. UnixFS is an abstraction layer that:
143
144 - Gives names to raw data blobs (so you can have `/foo/bar.txt` instead of just hashes)
145 - Represents directories as lists of named links to other nodes
146 - Organizes large files as trees of smaller chunks
147 - Makes these structures cryptographically verifiable - any tampering is detectable because it would change the hashes
148
149 With `--raw-leaves`, leaf nodes store raw data without the UnixFS wrapper. This is more efficient and is the default when using CIDv1.
150
151 ## Code Architecture
152
153 The add flow spans several layers:
154
155 ```mermaid
156 flowchart TD
157 subgraph CLI ["CLI Layer (kubo)"]
158 A["core/commands/add.go<br/>parses flags, shows progress"]
159 end
160 subgraph API ["CoreAPI Layer (kubo)"]
161 B["core/coreapi/unixfs.go<br/>UnixfsAPI.Add() entry point"]
162 end
163 subgraph Adder ["Adder (kubo)"]
164 C["core/coreunix/add.go<br/>orchestrates chunking, DAG building, MFS, pinning"]
165 end
166 subgraph Boxo ["boxo libraries"]
167 D["chunker/ - splits data into chunks"]
168 E["ipld/unixfs/ - DAG layout and UnixFS format"]
169 F["mfs/ - mutable filesystem abstraction"]
170 G["pinning/ - pin management"]
171 H["blockstore/ - block storage"]
172 end
173 A --> B --> C --> Boxo
174 ```
175
176 ### Key Files
177
178 | Component | Location |
179 |-----------|----------|
180 | CLI command | `core/commands/add.go` |
181 | API implementation | `core/coreapi/unixfs.go` |
182 | Adder logic | `core/coreunix/add.go` |
183 | Chunking | [boxo/chunker](https://github.com/ipfs/boxo/tree/main/chunker) |
184 | DAG layouts | [boxo/ipld/unixfs/importer](https://github.com/ipfs/boxo/tree/main/ipld/unixfs/importer) |
185 | MFS | [boxo/mfs](https://github.com/ipfs/boxo/tree/main/mfs) |
186 | Pinning | [boxo/pinning/pinner](https://github.com/ipfs/boxo/tree/main/pinning/pinner) |
187
188 ### The Adder
189
190 The `Adder` type in `core/coreunix/add.go` is the workhorse. It:
191
192 1. **Creates an MFS root** - temporary in-memory filesystem for building the DAG
193 2. **Processes files recursively** - chunks each file and builds DAG nodes
194 3. **Commits to blockstore** - persists all blocks
195 4. **Pins the result** - keeps content from being removed
196 5. **Returns the root CID**
197
198 Key methods:
199
200 - `AddAllAndPin()` - main entry point
201 - `addFileNode()` - handles a single file or directory
202 - `add()` - chunks data and builds the DAG using boxo's layout builders
203
204 ## Further Reading
205
206 - [UnixFS specification](https://specs.ipfs.tech/unixfs/)
207 - [IPLD and Merkle DAGs](https://docs.ipfs.tech/concepts/merkle-dag/)
208 - [Pinning](https://docs.ipfs.tech/concepts/persistence/)
209 - [MFS (Mutable File System)](https://docs.ipfs.tech/concepts/file-systems/#mutable-file-system-mfs)