@cryptotaxi247 / kubo / commits / 474b77a2b

importer: remove `UnixfsNode` from the balanced builder

The `UnixfsNode` structure has multiple pointers to many (non-complementary) mutually exclusive node types, only some of them are active (not-`nil`) at a given time in the code path which made the code too convoluted. Specifically, the most important distinction between node types was being hidden: leaf nodes vs internal (non-leaf) nodes. Remove entirely the use of `UnixfsNode` from the `balanced` package replacing it in turn with the newly created `FSNodeOverDag` structure that represents the UnixFS node encoded inside the DAG node, primarily used for internal node representations. Leaf nodes are handled exclusively in the `NewLeafDataNode` encapsulating its multiple representations (that we're previously exposed in `UnixfsNode` as conflicting pointers). The `builder.go` file has been completely rewritten, although the basic DAG creation algorithm has been preserved (extending a full DAG by creating a new root and linking the old one as its child), the most significant modification has been in the loop of `Layout` that now only handles internal nodes (i.e., nodes with `depth` bigger than zero) to be able to adapt `fillNodeRec` to only that scenario (avoiding the replace logic of the zero `depth` case with the defective `Set` function, now removed). The `fillNodeRec` now explicitly returns the `ipld.Node` and the size of the file data it's storing to propagate it upwards into the DAG. The `DagBuilderHelper` was heavily extended to incorporate `ipld.Node` functions that would replace the `UnixfsNode` ones used by the balanced builder: `NewLeafNode()`, `NewLeafDataNode()` and `AddNodeAndClose()`. Also, the `ProcessFileStore` function was incorporated to encapsulate all the logic related to the Filestore support which was scattered throughout the builder logic, the `offset` that was being passed through most functions is now a part of the `DagBuilderHelper`. This has turned out to be a rather big commit, it should have been split into more smaller and logically cohesive commits, but the `UnixfsNode` was too entangled inside the logic and that would have required a progressive modification of the `UnixfsNode` structure as well, which wasn't possible as it is still being used by the balanced builder (the same reason why most of the `UnixfsNode`-related functions cannot yet be removed, leaving the `helpers.go` file mostly untouched). License: MIT Signed-off-by: Lucas Molas <schomatis@gmail.com>

Lucas Molas committed Jun 17, 2018 at 10:51 UTC 474b77a2bdb1c15e73efa44e530a6f6f118a9312
3 files changed +433 -92
importer/balanced/builder.go
+217 -75
@@ -1,113 +1,255 @@
1 -// Package balanced provides methods to build balanced DAGs.
2 -// In a balanced DAG, nodes are added to a single root
3 -// until the maximum number of links is reached (with leaves
4 -// being at depth 0). Then, a new root is created, and points to the
5 -// old root, and incorporates a new child, which proceeds to be
6 -// filled up (link) to more leaves. In all cases, the Data (chunks)
7 -// is stored only at the leaves, with the rest of nodes only
8 -// storing links to their children.
9 -//
10 -// In a balanced DAG, nodes fill their link capacity before
11 -// creating new ones, thus depth only increases when the
12 -// current tree is completely full.
13 -//
14 -// Balanced DAGs are generalistic DAGs in which all leaves
15 -// are at the same distance from the root.
1 +// Package balanced provides methods to build balanced DAGs, which are generalistic
2 +// DAGs in which all leaves (nodes representing chunks of data) are at the same
3 +// distance from the root. Nodes can have only a maximum number of children; to be
4 +// able to store more leaf data nodes balanced DAGs are extended by increasing its
5 +// depth (and having more intermediary nodes).
6 +//
7 +// Internal nodes are always represented by UnixFS nodes (of type `File`) encoded
8 +// inside DAG nodes (see the `go-ipfs/unixfs` package for details of UnixFS). In
9 +// contrast, leaf nodes with data have multiple possible representations: UnixFS
10 +// nodes as above, raw nodes with just the file data (no format) and Filestore
11 +// nodes (that directly link to the file on disk using a format stored on a raw
12 +// node, see the `go-ipfs/filestore` package for details of Filestore.)
13 +//
14 +// In the case the entire file fits into just one node it will be formatted as a
15 +// (single) leaf node (without parent) with the possible representations already
16 +// mentioned. This is the only scenario where the root can be of a type different
17 +// that the UnixFS node.
18 +//
19 +// +-------------+
20 +// | Root 4 |
21 +// +-------------+
22 +// |
23 +// +--------------------------+----------------------------+
24 +// | |
25 +// +-------------+ +-------------+
26 +// | Node 2 | | Node 5 |
27 +// +-------------+ +-------------+
28 +// | |
29 +// +-------------+-------------+ +-------------+
30 +// | | |
31 +// +-------------+ +-------------+ +-------------+
32 +// | Node 1 | | Node 3 | | Node 6 |
33 +// +-------------+ +-------------+ +-------------+
34 +// | | |
35 +// +------+------+ +------+------+ +------+
36 +// | | | | |
37 +// +=========+ +=========+ +=========+ +=========+ +=========+
38 +// | Chunk 1 | | Chunk 2 | | Chunk 3 | | Chunk 4 | | Chunk 5 |
39 +// +=========+ +=========+ +=========+ +=========+ +=========+
40 +//
41 package balanced
42
43 import (
44 "errors"
45
46 h "github.com/ipfs/go-ipfs/importer/helpers"
47 + ft "github.com/ipfs/go-ipfs/unixfs"
48
49 ipld "gx/ipfs/QmWi2BYBL5gJ3CiAiQchg6rn1A8iBsrWy51EYxvHVjFvLb/go-ipld-format"
50 )
51
26 -// Layout builds a balanced DAG. Data is stored at the leaves
27 -// and depth only increases when the tree is full, that is, when
28 -// the root node has reached the maximum number of links.
52 +// Layout builds a balanced DAG layout. In a balanced DAG of depth 1, leaf nodes
53 +// with data are added to a single `root` until the maximum number of links is
54 +// reached. Then, to continue adding more data leaf nodes, a `newRoot` is created
55 +// pointing to the old `root` (which will now become and intermediary node),
56 +// increasing the depth of the DAG to 2. This will increase the maximum number of
57 +// data leaf nodes the DAG can have (`Maxlinks() ^ depth`). The `fillNodeRec`
58 +// function will add more intermediary child nodes to `newRoot` (which already has
59 +// `root` as child) that in turn will have leaf nodes with data added to them.
60 +// After that process is completed (the maximum number of links is reached),
61 +// `fillNodeRec` will return and the loop will be repeated: the `newRoot` created
62 +// will become the old `root` and a new root will be created again to increase the
63 +// depth of the DAG. The process is repeated until there is no more data to add
64 +// (i.e. the DagBuilderHelper’s Done() function returns true).
65 +//
66 +// The nodes are filled recursively, so the DAG is built from the bottom up. Leaf
67 +// nodes are created first using the chunked file data and its size. The size is
68 +// then bubbled up to the parent (internal) node, which aggregates all the sizes of
69 +// its children and bubbles that combined size up to its parent, and so on up to
70 +// the root. This way, a balanced DAG acts like a B-tree when seeking to a byte
71 +// offset in the file the graph represents: each internal node uses the file size
72 +// of its children as an index when seeking.
73 +//
74 +// `Layout` creates a root and hands it off to be filled:
75 +//
76 +// +-------------+
77 +// | Root 1 |
78 +// +-------------+
79 +// |
80 +// ( fillNodeRec fills in the )
81 +// ( chunks on the root. )
82 +// |
83 +// +------+------+
84 +// | |
85 +// + - - - - + + - - - - +
86 +// | Chunk 1 | | Chunk 2 |
87 +// + - - - - + + - - - - +
88 +//
89 +// ↓
90 +// When the root is full but there's more data...
91 +// ↓
92 +//
93 +// +-------------+
94 +// | Root 1 |
95 +// +-------------+
96 +// |
97 +// +------+------+
98 +// | |
99 +// +=========+ +=========+ + - - - - +
100 +// | Chunk 1 | | Chunk 2 | | Chunk 3 |
101 +// +=========+ +=========+ + - - - - +
102 +//
103 +// ↓
104 +// ...Layout's job is to create a new root.
105 +// ↓
106 +//
107 +// +-------------+
108 +// | Root 2 |
109 +// +-------------+
110 +// |
111 +// +-------------+ - - - - - - - - +
112 +// | |
113 +// +-------------+ ( fillNodeRec creates the )
114 +// | Node 1 | ( branch that connects )
115 +// +-------------+ ( "Root 2" to "Chunk 3." )
116 +// | |
117 +// +------+------+ + - - - - -+
118 +// | | |
119 +// +=========+ +=========+ + - - - - +
120 +// | Chunk 1 | | Chunk 2 | | Chunk 3 |
121 +// +=========+ +=========+ + - - - - +
122 +//
123 func Layout(db *h.DagBuilderHelper) (ipld.Node, error) {
30 - var offset uint64
31 - var root *h.UnixfsNode
32 - for level := 0; !db.Done(); level++ {
33 -
34 - nroot := db.NewUnixfsNode()
35 - db.SetPosInfo(nroot, 0)
36 -
37 - // add our old root as a child of the new root.
38 - if root != nil { // nil if it's the first node.
39 - if err := nroot.AddChild(root, db); err != nil {
40 - return nil, err
41 - }
42 - }
43 -
44 - // fill it up.
45 - if err := fillNodeRec(db, nroot, level, offset); err != nil {
46 - return nil, err
47 - }
48 -
49 - offset = nroot.FileSize()
50 - root = nroot
51 -
52 - }
53 - if root == nil {
54 - // this should only happen with an empty node, so return a leaf
55 - var err error
56 - root, err = db.NewLeaf(nil)
124 + if db.Done() {
125 + // No data, return just an empty node.
126 + root, err := db.NewLeafNode(nil)
127 if err != nil {
128 return nil, err
129 }
130 + // This works without Filestore support (`ProcessFileStore`).
131 + // TODO: Why? Is there a test case missing?
132 +
133 + return db.AddNodeAndClose(root)
134 }
135
62 - out, err := db.Add(root)
136 + // The first `root` will be a single leaf node with data
137 + // (corner case), after that subsequent `root` nodes will
138 + // always be internal nodes (with a depth > 0) that can
139 + // be handled by the loop.
140 + root, fileSize, err := db.NewLeafDataNode()
141 if err != nil {
142 return nil, err
143 }
144
67 - err = db.Close()
68 - if err != nil {
69 - return nil, err
145 + // Each time a DAG of a certain `depth` is filled (because it
146 + // has reached its maximum capacity of `db.Maxlinks()` per node)
147 + // extend it by making it a sub-DAG of a bigger DAG with `depth+1`.
148 + for depth := 1; !db.Done(); depth++ {
149 +
150 + // Add the old `root` as a child of the `newRoot`.
151 + newRoot := db.NewFSNodeOverDag(ft.TFile)
152 + newRoot.AddChild(root, fileSize, db)
153 +
154 + // Fill the `newRoot` (that has the old `root` already as child)
155 + // and make it the current `root` for the next iteration (when
156 + // it will become "old").
157 + root, fileSize, err = fillNodeRec(db, newRoot, depth)
158 + if err != nil {
159 + return nil, err
160 + }
161 }
162
72 - return out, nil
163 + return db.AddNodeAndClose(root)
164 }
165
75 -// fillNodeRec will fill the given node with data from the dagBuilders input
76 -// source down to an indirection depth as specified by 'depth'
77 -// it returns the total dataSize of the node, and a potential error
166 +// fillNodeRec will "fill" the given internal (non-leaf) `node` with data by
167 +// adding child nodes to it, either leaf data nodes (if `depth` is 1) or more
168 +// internal nodes with higher depth (and calling itself recursively on them
169 +// until *they* are filled with data). The data to fill the node with is
170 +// provided by DagBuilderHelper.
171 +//
172 +// `node` represents a (sub-)DAG root that is being filled. If called recursively,
173 +// it is `nil`, a new node is created. If it has been called from `Layout` (see
174 +// diagram below) it points to the new root (that increases the depth of the DAG),
175 +// it already has a child (the old root). New children will be added to this new
176 +// root, and those children will in turn be filled (calling `fillNodeRec`
177 +// recursively).
178 +//
179 +// +-------------+
180 +// | `node` |
181 +// | (new root) |
182 +// +-------------+
183 +// |
184 +// +-------------+ - - - - - - + - - - - - - - - - - - +
185 +// | | |
186 +// +--------------+ + - - - - - + + - - - - - +
187 +// | (old root) | | new child | | |
188 +// +--------------+ + - - - - - + + - - - - - +
189 +// | | |
190 +// +------+------+ + - - + - - - +
191 +// | | | |
192 +// +=========+ +=========+ + - - - - + + - - - - +
193 +// | Chunk 1 | | Chunk 2 | | Chunk 3 | | Chunk 4 |
194 +// +=========+ +=========+ + - - - - + + - - - - +
195 +//
196 +// The `node` to be filled uses the `FSNodeOverDag` abstraction that allows adding
197 +// child nodes without packing/unpacking the UnixFS layer node (having an internal
198 +// `ft.FSNode` cache).
199 +//
200 +// It returns the `ipld.Node` representation of the passed `node` filled with
201 +// children and the `nodeFileSize` with the total size of the file chunk (leaf)
202 +// nodes stored under this node (parent nodes store this to enable efficient
203 +// seeking through the DAG when reading data later).
204 //
205 // warning: **children** pinned indirectly, but input node IS NOT pinned.
80 -func fillNodeRec(db *h.DagBuilderHelper, node *h.UnixfsNode, depth int, offset uint64) error {
81 - if depth < 0 {
82 - return errors.New("attempt to fillNode at depth < 0")
206 +func fillNodeRec(db *h.DagBuilderHelper, node *h.FSNodeOverDag, depth int) (filledNode ipld.Node, nodeFileSize uint64, err error) {
207 + if depth < 1 {
208 + return nil, 0, errors.New("attempt to fillNode at depth < 1")
209 }
210
85 - // Base case
86 - if depth <= 0 { // catch accidental -1's in case error above is removed.
87 - child, err := db.GetNextDataNode()
88 - if err != nil {
89 - return err
90 - }
91 -
92 - node.Set(child)
93 - return nil
211 + if node == nil {
212 + node = db.NewFSNodeOverDag(ft.TFile)
213 }
214
96 - // while we have room AND we're not done
215 + // Child node created on every iteration to add to parent `node`.
216 + // It can be a leaf node or another internal node.
217 + var childNode ipld.Node
218 + // File size from the child node needed to update the `FSNode`
219 + // in `node` when adding the child.
220 + var childFileSize uint64
221 +
222 + // While we have room and there is data available to be added.
223 for node.NumChildren() < db.Maxlinks() && !db.Done() {
98 - child := db.NewUnixfsNode()
99 - db.SetPosInfo(child, offset)
224
101 - err := fillNodeRec(db, child, depth-1, offset)
102 - if err != nil {
103 - return err
225 + if depth == 1 {
226 + // Base case: add leaf node with data.
227 + childNode, childFileSize, err = db.NewLeafDataNode()
228 + if err != nil {
229 + return nil, 0, err
230 + }
231 + } else {
232 + // Recursion case: create an internal node to in turn keep
233 + // descending in the DAG and adding child nodes to it.
234 + childNode, childFileSize, err = fillNodeRec(db, nil, depth-1)
235 + if err != nil {
236 + return nil, 0, err
237 + }
238 }
239
106 - if err := node.AddChild(child, db); err != nil {
107 - return err
240 + err = node.AddChild(childNode, childFileSize, db)
241 + if err != nil {
242 + return nil, 0, err
243 }
109 - offset += child.FileSize()
244 }
245
112 - return nil
246 + nodeFileSize = node.FileSize()
247 +
248 + // Get the final `dag.ProtoNode` with the `FSNode` data encoded inside.
249 + filledNode, err = node.Commit()
250 + if err != nil {
251 + return nil, 0, err
252 + }
253 +
254 + return filledNode, nodeFileSize, nil
255 }
importer/helpers/dagbuilder.go
+216 -6
@@ -7,7 +7,9 @@ import (
7
8 dag "github.com/ipfs/go-ipfs/merkledag"
9 ft "github.com/ipfs/go-ipfs/unixfs"
10 + pb "github.com/ipfs/go-ipfs/unixfs/pb"
11
12 + pi "gx/ipfs/QmUWsXLvYYDAaoAt9TPZpFX4ffHHMg46AHrz1ZLTN5ABbe/go-ipfs-posinfo"
13 ipld "gx/ipfs/QmWi2BYBL5gJ3CiAiQchg6rn1A8iBsrWy51EYxvHVjFvLb/go-ipld-format"
14 chunker "gx/ipfs/QmXnzH7wowyLZy8XJxxaQCVTgLMcDXdMBznmsrmQWCyiQV/go-ipfs-chunker"
15 cid "gx/ipfs/QmapdYm1b22Frv3k17fqrBYTFRxwiaVJkB299Mfn33edeB/go-cid"
@@ -24,9 +26,21 @@ type DagBuilderHelper struct {
26 nextData []byte // the next item to return.
27 maxlinks int
28 batch *ipld.Batch
27 - fullPath string
28 - stat os.FileInfo
29 prefix *cid.Prefix
30 +
31 + // Filestore support variables.
32 + // ----------------------------
33 + // TODO: Encapsulate in `FilestoreNode` (which is basically what they are).
34 + //
35 + // Besides having the path this variable (if set) is used as a flag
36 + // to indicate that Filestore should be used.
37 + fullPath string
38 + stat os.FileInfo
39 + // Keeps track of the current file size added to the DAG (used in
40 + // the balanced builder). It is assumed that the `DagBuilderHelper`
41 + // is not reused to construct another DAG, but a new one (with a
42 + // zero `offset`) is created.
43 + offset uint64
44 }
45
46 // DagBuilderParams wraps configuration options to create a DagBuilderHelper
@@ -131,6 +145,11 @@ func (db *DagBuilderHelper) NewUnixfsNode() *UnixfsNode {
145 return n
146 }
147
148 +// GetPrefix returns the internal `cid.Prefix` set in the builder.
149 +func (db *DagBuilderHelper) GetPrefix() *cid.Prefix {
150 + return db.prefix
151 +}
152 +
153 // NewLeaf creates a leaf node filled with data. If rawLeaves is
154 // defined than a raw leaf will be returned. Otherwise, if data is
155 // nil the type field will be TRaw (for backwards compatibility), if
@@ -166,6 +185,44 @@ func (db *DagBuilderHelper) NewLeaf(data []byte) (*UnixfsNode, error) {
185 return blk, nil
186 }
187
188 +// NewLeafNode is a variation from `NewLeaf` (see its description) that
189 +// returns an `ipld.Node` instead.
190 +func (db *DagBuilderHelper) NewLeafNode(data []byte) (ipld.Node, error) {
191 + if len(data) > BlockSizeLimit {
192 + return nil, ErrSizeLimitExceeded
193 + }
194 +
195 + if db.rawLeaves {
196 + // Encapsulate the data in a raw node.
197 + if db.prefix == nil {
198 + return dag.NewRawNode(data), nil
199 + }
200 + rawnode, err := dag.NewRawNodeWPrefix(data, *db.prefix)
201 + if err != nil {
202 + return nil, err
203 + }
204 + return rawnode, nil
205 + }
206 +
207 + // Encapsulate the data in UnixFS node (instead of a raw node).
208 + fsNodeOverDag := db.NewFSNodeOverDag(ft.TFile)
209 + fsNodeOverDag.SetFileData(data)
210 + node, err := fsNodeOverDag.Commit()
211 + if err != nil {
212 + return nil, err
213 + }
214 + // TODO: Encapsulate this sequence of calls into a function that
215 + // just returns the final `ipld.Node` avoiding going through
216 + // `FSNodeOverDag`.
217 + // TODO: Using `TFile` for backwards-compatibility, a bug in the
218 + // balanced builder was causing the leaf nodes to be generated
219 + // with this type instead of `TRaw`, the one that should be used
220 + // (like the trickle builder does).
221 + // (See https://github.com/ipfs/go-ipfs/pull/5120.)
222 +
223 + return node, nil
224 +}
225 +
226 // newUnixfsBlock creates a new Unixfs node to represent a raw data block
227 func (db *DagBuilderHelper) newUnixfsBlock() *UnixfsNode {
228 n := &UnixfsNode{
@@ -211,12 +268,63 @@ func (db *DagBuilderHelper) GetNextDataNode() (*UnixfsNode, error) {
268 return db.NewLeaf(data)
269 }
270
214 -// SetPosInfo sets the offset information of a node using the fullpath and stat
215 -// from the DagBuilderHelper.
216 -func (db *DagBuilderHelper) SetPosInfo(node *UnixfsNode, offset uint64) {
271 +// NewLeafDataNode is a variation of `GetNextDataNode` that returns
272 +// an `ipld.Node` instead. It builds the `node` with the data obtained
273 +// from the Splitter and returns it with the `dataSize` (that will be
274 +// used to keep track of the DAG file size). The size of the data is
275 +// computed here because after that it will be hidden by `NewLeafNode`
276 +// inside a generic `ipld.Node` representation.
277 +func (db *DagBuilderHelper) NewLeafDataNode() (node ipld.Node, dataSize uint64, err error) {
278 + fileData, err := db.Next()
279 + if err != nil {
280 + return nil, 0, err
281 + }
282 + dataSize = uint64(len(fileData))
283 +
284 + // Create a new leaf node containing the file chunk data.
285 + node, err = db.NewLeafNode(fileData)
286 + if err != nil {
287 + return nil, 0, err
288 + }
289 +
290 + // Convert this leaf to a `FilestoreNode` if needed.
291 + node = db.ProcessFileStore(node, dataSize)
292 +
293 + return node, dataSize, nil
294 +}
295 +
296 +// ProcessFileStore generates, if Filestore is being used, the
297 +// `FilestoreNode` representation of the `ipld.Node` that
298 +// contains the file data. If Filestore is not being used just
299 +// return the same node to continue with its addition to the DAG.
300 +//
301 +// The `db.offset` is updated at this point (instead of when
302 +// `NewLeafDataNode` is called, both work in tandem but the
303 +// offset is more related to this function).
304 +func (db *DagBuilderHelper) ProcessFileStore(node ipld.Node, dataSize uint64) ipld.Node {
305 + // Check if Filestore is being used.
306 if db.fullPath != "" {
218 - node.SetPosInfo(offset, db.fullPath, db.stat)
307 + // Check if the node is actually a raw node (needed for
308 + // Filestore support).
309 + if _, ok := node.(*dag.RawNode); ok {
310 + fn := &pi.FilestoreNode{
311 + Node: node,
312 + PosInfo: &pi.PosInfo{
313 + Offset: db.offset,
314 + FullPath: db.fullPath,
315 + Stat: db.stat,
316 + },
317 + }
318 +
319 + // Update `offset` with the size of the data generated by `db.Next`.
320 + db.offset += dataSize
321 +
322 + return fn
323 + }
324 }
325 +
326 + // Filestore is not used, return the same `node` argument.
327 + return node
328 }
329
330 // Add sends a node to the DAGService, and returns it.
@@ -246,3 +354,105 @@ func (db *DagBuilderHelper) Maxlinks() int {
354 func (db *DagBuilderHelper) Close() error {
355 return db.batch.Commit()
356 }
357 +
358 +// AddNodeAndClose adds the last `ipld.Node` from the DAG and
359 +// closes the builder. It returns the same `node` passed as
360 +// argument.
361 +func (db *DagBuilderHelper) AddNodeAndClose(node ipld.Node) (ipld.Node, error) {
362 + err := db.batch.Add(node)
363 + if err != nil {
364 + return nil, err
365 + }
366 +
367 + err = db.Close()
368 + if err != nil {
369 + return nil, err
370 + }
371 +
372 + return node, nil
373 +}
374 +
375 +// FSNodeOverDag encapsulates an `unixfs.FSNode` that will be stored in a
376 +// `dag.ProtoNode`. Instead of just having a single `ipld.Node` that
377 +// would need to be constantly (un)packed to access and modify its
378 +// internal `FSNode` in the process of creating a UnixFS DAG, this
379 +// structure stores an `FSNode` cache to manipulate it (add child nodes)
380 +// directly , and only when the node has reached its final (immutable) state
381 +// (signaled by calling `Commit()`) is it committed to a single (indivisible)
382 +// `ipld.Node`.
383 +//
384 +// It is used mainly for internal (non-leaf) nodes, and for some
385 +// representations of data leaf nodes (that don't use raw nodes or
386 +// Filestore).
387 +//
388 +// It aims to replace the `UnixfsNode` structure which encapsulated too
389 +// many possible node state combinations.
390 +//
391 +// TODO: Revisit the name.
392 +type FSNodeOverDag struct {
393 + dag *dag.ProtoNode
394 + file *ft.FSNode
395 +}
396 +
397 +// NewFSNodeOverDag creates a new `dag.ProtoNode` and `ft.FSNode`
398 +// decoupled from one onther (and will continue in that way until
399 +// `Commit` is called), with `fsNodeType` specifying the type of
400 +// the UnixFS layer node (either `File` or `Raw`).
401 +func (db *DagBuilderHelper) NewFSNodeOverDag(fsNodeType pb.Data_DataType) *FSNodeOverDag {
402 + node := new(FSNodeOverDag)
403 + node.dag = new(dag.ProtoNode)
404 + node.dag.SetPrefix(db.GetPrefix())
405 +
406 + node.file = ft.NewFSNode(fsNodeType)
407 +
408 + return node
409 +}
410 +
411 +// AddChild adds a `child` `ipld.Node` to both node layers. The
412 +// `dag.ProtoNode` creates a link to the child node while the
413 +// `ft.FSNode` stores its file size (that is, not the size of the
414 +// node but the size of the file data that it is storing at the
415 +// UnixFS layer). The child is also stored in the `DAGService`.
416 +func (n *FSNodeOverDag) AddChild(child ipld.Node, fileSize uint64, db *DagBuilderHelper) error {
417 + err := n.dag.AddNodeLink("", child)
418 + if err != nil {
419 + return err
420 + }
421 +
422 + n.file.AddBlockSize(fileSize)
423 +
424 + return db.batch.Add(child)
425 +}
426 +
427 +// Commit unifies (resolves) the cache nodes into a single `ipld.Node`
428 +// that represents them: the `ft.FSNode` is encoded inside the
429 +// `dag.ProtoNode`.
430 +//
431 +// TODO: Evaluate making it read-only after committing.
432 +func (n *FSNodeOverDag) Commit() (ipld.Node, error) {
433 + fileData, err := n.file.GetBytes()
434 + if err != nil {
435 + return nil, err
436 + }
437 + n.dag.SetData(fileData)
438 +
439 + return n.dag, nil
440 +}
441 +
442 +// NumChildren returns the number of children of the `ft.FSNode`.
443 +func (n *FSNodeOverDag) NumChildren() int {
444 + return n.file.NumChildren()
445 +}
446 +
447 +// FileSize returns the `Filesize` attribute from the underlying
448 +// representation of the `ft.FSNode`.
449 +func (n *FSNodeOverDag) FileSize() uint64 {
450 + return n.file.FileSize()
451 +}
452 +
453 +// SetFileData stores the `fileData` in the `ft.FSNode`. It
454 +// should be used only when `FSNodeOverDag` represents a leaf
455 +// node (internal nodes don't carry data, just file sizes).
456 +func (n *FSNodeOverDag) SetFileData(fileData []byte) {
457 + n.file.SetData(fileData)
458 +}
importer/helpers/helpers.go
-11
@@ -70,17 +70,6 @@ func (n *UnixfsNode) NumChildren() int {
70 return n.ufmt.NumChildren()
71 }
72
73 -// Set replaces the current UnixfsNode with another one. It performs
74 -// a shallow copy.
75 -func (n *UnixfsNode) Set(other *UnixfsNode) {
76 - n.node = other.node
77 - n.raw = other.raw
78 - n.rawnode = other.rawnode
79 - if other.ufmt != nil {
80 - n.ufmt.SetData(other.ufmt.Data())
81 - }
82 -}
83 -
73 // GetChild gets the ith child of this node from the given DAGService.
74 func (n *UnixfsNode) GetChild(ctx context.Context, i int, ds ipld.DAGService) (*UnixfsNode, error) {
75 nd, err := n.node.Links()[i].GetNode(ctx, ds)