Cleanup and resolve docs
- Cleaned up some of the language. - Added versioned links to the code. - Provided examples License: MIT Signed-off-by: Nitish Malhotra <nitish.malhotra@gmail.com>
nmalhotra committed
Jan 6, 2019 at 18:48 UTC
e229bc27c0472b1673bee308b3364e92975ce371
1 file changed
+63
-61
docs/add-code-flow.md
+63
-61
@@ -1,97 +1,99 @@
1
-https://github.com/ipfs/go-ipfs/blob/master/core/commands/add.go#L208-L213
1
+# IPFS : The `Add` command demystified
2
+
3
+https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/commands/add.go#L208-L213
4
5
The goal of this document is to capture the code flow for adding a file (see the `coreapi` package) using the IPFS CLI, in the process exploring some datastructures and packages like `ipld.Node` (aka `dagnode`, `FSNode`, `MFS`, etc.
6
5
-# `UnixfsAPI.Add()`
6
-*Entrypoint into the `Unixfs` package*
7
+**Try this yourself**
8
+>
9
+> ```
10
+> # Convert a file to the IPFS format.
11
+> echo "Hello World" > new-file
12
+> ipfs add new-file
13
+> added QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy47CnJDgvs8u new-file
14
+> 12 B / 12 B [=========================================================] 100.00%
15
+>
16
+> # Add a file to the MFS.
17
+> NEW_FILE_HASH=$(ipfs add new-file -Q)
18
+> ipfs files cp /ipfs/$NEW_FILE_HASH /new-file
19
+>
20
+> # Get information from the file in MFS.
21
+> ipfs files stat /new-file
22
+> # QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy47CnJDgvs8u
23
+> # Size: 12
24
+> # CumulativeSize: 20
25
+> # ChildBlocks: 0
26
+> # Type: file
27
+>
28
+> # Retrieve the contents.
29
+> ipfs files read /new-file
30
+> # Hello World
31
+> ```
32
+
33
+---
34
8
-https://github.com/ipfs/go-ipfs/blob/master/core/coreapi/unixfs.go#L78-L86
35
+# **[`UnixfsAPI.Add()`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreapi/unixfs.go#L31)** - *Entrypoint into the `Unixfs` package*
36
37
The `UnixfsAPI.Add()` acts on the input data or files, to build a _merkledag_ node (in essence it is the entire tree represented by the root node) and adds it to the _blockstore_.
38
Within the function, a new `Adder` is created with the configured `Blockstore` and __DAG service__`.
39
40
+## **[`adder.AddAllAndPin(files)`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L403)** - *Entrypoint to the `Add` logic*
41
+ encapsulates a lot of the underlying functionality that will be investigated in the following sections.
42
14
-# `Adder.AddAllAndPin(files.File)`
15
-*Entrypoint*
16
-
17
-https://github.com/ipfs/go-ipfs/blob/master/core/coreunix/add.go#L427-L431
18
-
19
-The more interesting stuff happens in the `Adder.AddAllAndPin(files)` function.
20
-
21
-https://github.com/ipfs/go-ipfs/blob/master/core/coreunix/add.go#L467-L476
43
+Our focus will be on the simplest case, a single file, handled by `Adder.addFileNode(path, file)` handled by `Adder.addFile(pathm files.File)`.
44
23
-Let's focus on the simple case of a single file, handled by `Adder.addFileNode(path, file)` which redirects the case to `Adder.addFile(pathm files.File)`.
45
+- **[`adder.addFile(path, files.File)`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L450)** - *Create the _DAG_ and add to `MFS`*
46
25
-## `Adder.addFile(path, files.File)`
26
-*Create the _DAG_ and add to `MFS`*
47
+ The `addFile(file)` method takes the data and converts it into a __DAG__ tree and adds the root of the tree in to the `MFS`.
48
28
-The `addFile(file)` method is responsible taking the data and converting it into a __DAG__ tree, followed by adding the root of the DAG tree in to the `MFS`.
49
+ https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L508-L521
50
30
-https://github.com/ipfs/go-ipfs/blob/master/core/coreunix/add.go#L508-L521
51
+ There are two main methods to focus on -
52
32
-There are two main methods to focus on -
53
+ 1. **[`adder.add(io.Reader)`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L115)** - *Create and return the **root** __DAG__ node*
54
34
-### `Adder.add(io.Reader)`
35
-*Create and return the **root** __DAG__ node*
55
+ This method converts the input data (`io.Reader`) to a __DAG__ tree, by splitting the data into _chunks_ using the `Chunker` and organizing them in to a __DAG__ (with a *trickle* or *balanced* layout. See [balanced](https://github.com/ipfs/go-unixfs/blob/6b769632e7eb8fe8f302e3f96bf5569232e7a3ee/importer/balanced/builder.go) for more info).
56
37
-https://github.com/ipfs/go-ipfs/blob/master/core/coreunix/add.go#L115-L137
57
+ The method returns the **root** `ipld.Node` of the __DAG__.
58
39
-This method converts the input _data_ (`io.Reader`) to a `DAG` tree. This is done by splitting the data into _chunks_ using the `Chunker` **(HELP: elaborate on chunker types ?)** and organizing them in a `DAG` (with a *trickle* or *balanced* layout). The method returns the **root** of the __DAG__, formatted as an `ipld.Node`.
59
+ 2. **[`adder.addNode(ipld.Node, path)`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L366)** - *Add **root** __DAG__ node to the `MFS`*
60
41
-### `Adder.addNode(ipld.Node, path)`
42
-*Add **root** __DAG__ node to the `MFS`*
61
+ Now that we have the **root** node of the `DAG`, this needs to be added to the `MFS` file system.
62
+ Fetch (or create, if doesn't already exist) the `MFS` **root** using `mfsRoot()`.
63
44
-https://github.com/ipfs/go-ipfs/blob/master/core/coreunix/add.go#L365-L399
64
+ > NOTE: The `MFS` **root** is an ephemeral root, created and destroyed solely for the `add` functionality.
65
46
-Now that we have the **root** node of the `DAG`, this needs to be added to the `MFS` file system.
47
-The `MFS` **root** is first fetched (or created, if doesn't already exist) by invoking `mfsRoot()`.
48
-Assuming the directory structure already exists in the MFS file system, (if it doesn't exist it will be created using `mfs.Mkdir()` function by passing in the `MFS` **root**), the **root** __DAG__ node is added to the `MFS` File system within the `mfs.PutNode()` function.
66
+ Assuming the directory already exists in the MFS file system, (if it doesn't exist it will be created using `mfs.Mkdir()`), the **root** __DAG__ node is added to the `MFS` File system using the `mfs.PutNode()` function.
67
50
-#### `[MFS] PutNode(mfs.Root, path, ipld.Node)`
51
-*Insert node at path into given `MFS`*
68
+ - **[MFS] [`PutNode(mfs.Root, path, ipld.Node)`](https://github.com/ipfs/go-mfs/blob/master/ops.go#L101)** - *Insert node at path into given `MFS`*
69
53
-https://github.com/ipfs/go-mfs/blob/master/ops.go#L101-L113
70
+ The `path` param is used to determine the `MFS Directory`, which is first looked up in the `MFS` using `lookupDir()` function. This is followed by adding the **root** __DAG__ node (`ipld.Node`) in to this `Directory` using `directory.AddChild()` method.
71
55
-In this the path is used to determine the `MFS` `Directory`, which is first lookup up in the `MFS` using `lookupDir()` function. This is followed by adding the **root** __DAG__ node (`ipld.Node`) in to the found `Directory` using `directory.AddChild()` method.
72
+ - **[MFS] Add Child To `UnixFS`**
73
+ - **[`directory.AddChild(filename, ipld.Node)`](https://github.com/ipfs/go-mfs/blob/master/dir.go#L374)** - *Add **root** __DAG__ node under this directory*
74
57
-#### - `directory.AddChild(filename, ipld.Node)`
58
-*Add **root** __DAG__ node , as filename, under this directory*
75
+ Within this method the node is added to the `Directory`'s __DAG service__ using the `dserv.Add()` method, followed by adding the **root** __DAG__ node with the given name, in the `directory.addUnixFSChild(directory.child{name, ipld.Node})` method.
76
60
-https://github.com/ipfs/go-mfs/blob/master/dir.go#L381-L402
77
+ - **[MFS] [`directory.addUnixFSChild(child)`](https://github.com/ipfs/go-mfs/blob/master/dir.go#L374)** - *Add child to inner UnixFS Directory*
78
62
-Within this method the node is added to the __DAG service__ of the `Directory` object using the `dserv.Add()` method [HELP NEEDED].
63
-This is subsequently followed by adding the **root** __DAG__ node by creating a `directory.child{}` object with the given name, within in the `directory.addUnixFSChild(directory.child{name, ipld.Node})` method.
79
+ The node is then added as a child to the inner `UnixFS` directory using the `(BasicDirectory).AddChild()` method.
80
65
-#### -- `directory.addUnixFSChild(child)`
66
-*Switch to HAMT (if configured) and add child to inner UnixFS Directory*
81
+ > NOTE: This is not to be confused with the `directory.AddChild(filename, ipld.Node)`, as this operates on the `UnixFS` `BasicDirectory` object only.
82
68
-https://github.com/ipfs/go-mfs/blob/master/dir.go#L406-L425
83
+ - **[UnixFS] [`(BasicDirectory).AddChild(ctx, name, ipld.Node)`](https://github.com/ipfs/go-unixfs/blob/master/io/directory.go#L142)** - *Add child to `BasicDirectory`*
84
70
-Here the transition of the UnixFS directory to __HAMT__ implemetation is done, if configured, wherein if the directory is of type `BasicDirectory`, it is converted to a __HAMT__ implementation.
71
-The node is then added as a child to the inner `UnixFS` directory using the `directory.AddChild()` method.
72
-Note: This is not to be confused with the `directory.AddChild(filename, ipld.Node)`, as this operates on the inner `UnixFS` `Directory` object only.
85
+ > IMPORTANT: It should be noted that the `BasicDirectory` struct of the `UnixFS` package, encasulates a node object of type `ProtoNode`, which is a different format from the `ipld.Node` we have been working on throughout this document.
86
74
-#### --- (inner)`Directory.AddChild(ctx, name, ipld.Node)`
87
+ This method first attempts to remove any old links (`ProtoNode.RemoveNodeLink(name)`) to the `ProtoNode` prior to adding a link to the newly added `ipld.Node`, using `ProtoNode.AddNodeLink(name, ipld.Node)`.
88
76
-This method vastly differs based on the inner `Directory` implementation (Basic vs HAMT). Let's focus on the `BasicDirectory` implementation to keep things simple.
89
+ - **[Merkledag] [`AddNodeLink()`](https://github.com/ipfs/go-unixfs/blob/master/io/directory.go#L142)**
90
78
-https://github.com/ipfs/go-unixfs/blob/master/io/directory.go#L142-L147
91
+ The `AddNodeLink()` method is where an `ipld.Link` is created with the `ipld.Node`'s `CID` and size in the `ipld.MakeLink(ipld.Node)` method, and is then appended to the `ProtoNode`'s links in the `ProtoNode.AddRawLink(name)` method.
92
80
-> IMPORTANT
81
-> It should be noted that the inner `Directory` of the `UnixFS` package, encasulates a node object of type `ProtoNode`, which is a different format as compared to the `ipld.Node` we have been working on throughout this document.
82
-
83
-This method first attempts to remove any old links (`ProtoNode.RemoveNodeLink(name)`) to the `ProtoNode` prior to adding a link to the newly added `ipld/Node`, using `ProtoNode.AddNodeLink(name, ipld.Node)`.
84
-
85
-https://github.com/ipfs/go-merkledag/blob/master/node.go#L99-L112
86
-
87
-The `AddNodeLink()` method is where an `ipld.Link` is created with the `ipld.Node`'s `CID` and size in the `ipld.MakeLink(ipld.Node)` method, and is then appended to the `ProtoNode`'s links in the `ProtoNode.AddRawLink(name)` method.
88
-
89
----
93
+- **[`adder.Finalize()`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L200)** - *Fetch and return the __DAG__ **root** from the `MFS` and `UnixFS` directory*
94
91
-## `adder.Finalize()`
92
-*Fetch and return the __DAG__ **root** from the `MFS` and `UnixFS` directory*
95
+ The `Finalize` method returns the `ipld.Node` from the `UnixFS` `Directory`.
96
94
-https://github.com/ipfs/go-ipfs/blob/master/core/coreunix/add.go#L199-L244
97
+- **[`adder.PinRoot()`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L171)** - *Pin all files under the `MFS` **root***
98
96
-The whole process ends with `adder.Finalize()` which returns the `ipld.Node` from the `UnixFS` `Directory`.
97
-**(HELP: Do we need to elaborate ?)**
\ No newline at end of file
99
+ The whole process ends with `PinRoot` recursively pinning all the files under the `MFS` **root**
\ No newline at end of file