@cryptotaxi247 / kubo / commits / 1bc15a4b4

docs: Tutorial - Using go-ipfs as a Library

David Dias committed Oct 29, 2019 at 09:12 UTC 1bc15a4b421cc24da2d06cdf7b8ee8ca1cbdf01c
6 files changed +445
.gitignore
+1
@@ -25,3 +25,4 @@ bin/cover
25 vendor
26 .tarball
27 go-ipfs-source.tar.gz
28 +docs/examples/go-ipfs-as-a-library/example-folder/Qm*
docs/examples/go-ipfs-as-a-library/README.md new
+113
@@ -0,0 +1,113 @@
1 +# Use go-ipfs as a library to spawn a node and add a file
2 +
3 +> This tutorial is the sibling of the [js-ipfs IPFS 101 tutorial](https://github.com/ipfs/js-ipfs/tree/master/examples/ipfs-101).
4 +
5 +By the end of this tutorial, you will learn how to:
6 +
7 +- Spawn an IPFS node that runs in process (no separate daemon process)
8 +- Create an IPFS repo
9 +- Add files and directories to IPFS
10 +- Retrieve those files and directories using ``cat`` and ``get``
11 +- Connect to other nodes in the network
12 +- Retrieve a file that only exists on the network
13 +- The difference between a node in DHT client mode and full DHT mode
14 +
15 +All of this using only golang!
16 +
17 +In order to complete this tutorial, you will need:
18 +- golang installed on your machine. See how at https://golang.org/doc/install
19 +- git installed on your machine (so that go can download the repo and the necessary dependencies). See how at https://git-scm.com/downloads
20 +- IPFS Desktop (for convenience) installed and running on your machine. See how at https://github.com/ipfs-shipyard/ipfs-desktop#ipfs-desktop
21 +
22 +
23 +**Disclaimer**: The example code is quite large (more than 300 lines of code) and it has been a great way to understand the scope of the [go-ipfs Core API](https://godoc.org/github.com/ipfs/interface-go-ipfs-core), and how it can be improved to further the user experience. You can expect to be able to come back to this example in the future and see how the number of lines of code have decreased and how the example have become simpler, making other go-ipfs programs simpler as well.
24 +
25 +## Getting started
26 +
27 +**Note:** Make sure you have [![](https://img.shields.io/badge/go-%3E%3D1.13.0-blue.svg?style=flat-square)](https://golang.org/dl/) installed.
28 +
29 +Download go-ipfs and jump into the example folder:
30 +
31 +```
32 +> go get -u github.com/ipfs/go-ipfs
33 +cd $GOPATH/src/github.com/ipfs/go-ipfs/docs/examples/go-ipfs-as-a-library
34 +```
35 +
36 +## Running the example as-is
37 +
38 +To run the example, simply do:
39 +
40 +```
41 +> go run main.go
42 +```
43 +
44 +You should see the following as output:
45 +
46 +```
47 +-- Getting an IPFS node running --
48 +Spawning node on a temporary repo
49 +IPFS node is running
50 +
51 +-- Adding and getting back files & directories --
52 +Added file to IPFS with CID /ipfs/QmV9tSDx9UiPeWExXEeH6aoDvmihvx6jD5eLb4jbTaKGps
53 +Added directory to IPFS with CID /ipfs/QmdQdu1fkaAUokmkfpWrmPHK78F9Eo9K2nnuWuizUjmhyn
54 +Got file back from IPFS (IPFS path: /ipfs/QmV9tSDx9UiPeWExXEeH6aoDvmihvx6jD5eLb4jbTaKGps) and wrote it to ./example-folder/QmV9tSDx9UiPeWExXEeH6aoDvmihvx6jD5eLb4jbTaKGps
55 +Got directory back from IPFS (IPFS path: /ipfs/QmdQdu1fkaAUokmkfpWrmPHK78F9Eo9K2nnuWuizUjmhyn) and wrote it to ./example-folder/QmdQdu1fkaAUokmkfpWrmPHK78F9Eo9K2nnuWuizUjmhyn
56 +
57 +-- Going to connect to a few nodes in the Network as bootstrappers --
58 +Fetching a file from the network with CID QmUaoioqU7bxezBQZkUcgcSyokatMY71sxsALxQmRRrHrj
59 +Wrote the file to ./example-folder/QmUaoioqU7bxezBQZkUcgcSyokatMY71sxsALxQmRRrHrj
60 +
61 +All done! You just finalized your first tutorial on how to use go-ipfs as a library
62 +```
63 +
64 +## Understanding the example
65 +
66 +In this example, we add a file and a directory with files; we get them back from IPFS; and then we use the IPFS network to fetch a file that we didn't have in our machines before.
67 +
68 +Each section below has links to lines of code in the file [main.go](./main.go). The code itself will have comments explaining what is happening for you.
69 +
70 +### The `func main() {}`
71 +
72 +The [main function](./main.go#L202-L331) is where the magic starts, and it is the best place to follow the path of what is happening in the tutorial.
73 +
74 +### Part 1: Getting an IPFS node running
75 +
76 +To get [get a node running](./main.go#L218-L223) as an [ephemeral node](./main.go#L114-L128) (that will cease to exist when the run ends), you will need to:
77 +
78 +- [Prepare and set up the plugins](./main.go#L30-L47)
79 +- [Create an IPFS repo](./main.go#L49-L68)
80 +- [Construct the IPFS node instance itself](./main.go#L72-L96)
81 +
82 +As soon as you construct the IPFS node instance, the node will be running.
83 +
84 +### Part 2: Adding a file and a directory to IPFS
85 +
86 +- [Prepare the file to be added to IPFS](./main.go#L166-L184))
87 +- [Add the file to IPFS](./main.go#L240-L243))
88 +- [Prepare the directory to be added to IPFS](./main.go#L186-L198))
89 +- [Add the directory to IPFS](./main.go#L252-L255))
90 +
91 +### Part 3: Getting the file and directory you added back
92 +
93 +- [Get the file back](./main.go#L265-L268))
94 +- [Write the file to your local filesystem](./main.go#L270-L273))
95 +- [Get the directory back](./main.go#L277-L280))
96 +- [Write the directory to your local filesystem](./main.go#L282-L285))
97 +
98 +### Part 4: Getting a file from the IPFS network
99 +
100 +- [Connect to nodes in the network](./main.go#L293-L310))
101 +- [Get the file from the network](./main.go#L318-L321))
102 +- [Write the file to your local filesystem](./main.go#L323-L326))
103 +
104 +### Bonus: Spawn a daemon on your existing IPFS repo (on the default path ~/.ipfs)
105 +
106 +As a bonus, you can also find lines that show you how to spawn a node over your default path (~/.ipfs) in case you had already started a node there before. To try it:
107 +
108 +- [Comment these lines](./main.go#L219-L223))
109 +- [Uncomment these lines](./main.go#L209-L216))
110 +
111 +## Voilá! You are now a go-ipfs hacker
112 +
113 +You've learned how to spawn a go-ipfs node using the go-ipfs core API. There are many more [methods to experiment next](https://godoc.org/github.com/ipfs/interface-go-ipfs-core). Happy hacking!
docs/examples/go-ipfs-as-a-library/example-folder/ipfs.paper.draft3.pdf
Binary files /dev/null and b/docs/examples/go-ipfs-as-a-library/example-folder/ipfs.paper.draft3.pdf differ
docs/examples/go-ipfs-as-a-library/example-folder/test-dir/ipfs-logo.png
Binary files /dev/null and b/docs/examples/go-ipfs-as-a-library/example-folder/test-dir/ipfs-logo.png differ
docs/examples/go-ipfs-as-a-library/example-folder/test-dir/ipfs.paper.draft3.pdf
Binary files /dev/null and b/docs/examples/go-ipfs-as-a-library/example-folder/test-dir/ipfs.paper.draft3.pdf differ
docs/examples/go-ipfs-as-a-library/main.go new
+331
@@ -0,0 +1,331 @@
1 +package main
2 +
3 +import (
4 + "context"
5 + "fmt"
6 + "io/ioutil"
7 + "log"
8 + "os"
9 + "path/filepath"
10 + "strings"
11 + "sync"
12 +
13 + config "github.com/ipfs/go-ipfs-config"
14 + files "github.com/ipfs/go-ipfs-files"
15 + libp2p "github.com/ipfs/go-ipfs/core/node/libp2p"
16 + icore "github.com/ipfs/interface-go-ipfs-core"
17 + icorepath "github.com/ipfs/interface-go-ipfs-core/path"
18 + peerstore "github.com/libp2p/go-libp2p-peerstore"
19 + ma "github.com/multiformats/go-multiaddr"
20 +
21 + "github.com/ipfs/go-ipfs/core"
22 + "github.com/ipfs/go-ipfs/core/coreapi"
23 + "github.com/ipfs/go-ipfs/plugin/loader" // This package is needed so that all the preloaded plugins are loaded automatically
24 + "github.com/ipfs/go-ipfs/repo/fsrepo"
25 + "github.com/libp2p/go-libp2p-core/peer"
26 +)
27 +
28 +/// ------ Setting up the IPFS Repo
29 +
30 +func setupPlugins(externalPluginsPath string) error {
31 + // Load any external plugins if available on externalPluginsPath
32 + plugins, err := loader.NewPluginLoader(filepath.Join(externalPluginsPath, "plugins"))
33 + if err != nil {
34 + return fmt.Errorf("error loading plugins: %s", err)
35 + }
36 +
37 + // Load preloaded and external plugins
38 + if err := plugins.Initialize(); err != nil {
39 + return fmt.Errorf("error initializing plugins: %s", err)
40 + }
41 +
42 + if err := plugins.Inject(); err != nil {
43 + return fmt.Errorf("error initializing plugins: %s", err)
44 + }
45 +
46 + return nil
47 +}
48 +
49 +func createTempRepo(ctx context.Context) (string, error) {
50 + repoPath, err := ioutil.TempDir("", "ipfs-shell")
51 + if err != nil {
52 + return "", fmt.Errorf("failed to get temp dir: %s", err)
53 + }
54 +
55 + // Create a config with default options and a 2048 bit key
56 + cfg, err := config.Init(ioutil.Discard, 2048)
57 + if err != nil {
58 + return "", err
59 + }
60 +
61 + // Create the repo with the config
62 + err = fsrepo.Init(repoPath, cfg)
63 + if err != nil {
64 + return "", fmt.Errorf("failed to init ephemeral node: %s", err)
65 + }
66 +
67 + return repoPath, nil
68 +}
69 +
70 +/// ------ Spawning the node
71 +
72 +// Creates an IPFS node and returns its coreAPI
73 +func createNode(ctx context.Context, repoPath string) (icore.CoreAPI, error) {
74 + // Open the repo
75 + repo, err := fsrepo.Open(repoPath)
76 + if err != nil {
77 + return nil, err
78 + }
79 +
80 + // Construct the node
81 +
82 + nodeOptions := &core.BuildCfg{
83 + Online: true,
84 + Routing: libp2p.DHTOption, // This option sets the node to be a full DHT node (both fetching and storing DHT Records)
85 + // Routing: libp2p.DHTClientOption, // This option sets the node to be a client DHT node (only fetching records)
86 + Repo: repo,
87 + }
88 +
89 + node, err := core.NewNode(ctx, nodeOptions)
90 + if err != nil {
91 + return nil, err
92 + }
93 +
94 + // Attach the Core API to the constructed node
95 + return coreapi.NewCoreAPI(node)
96 +}
97 +
98 +// Spawns a node on the default repo location, if the repo exists
99 +func spawnDefault(ctx context.Context) (icore.CoreAPI, error) {
100 + defaultPath, err := config.PathRoot()
101 + if err != nil {
102 + // shouldn't be possible
103 + return nil, err
104 + }
105 +
106 + if err := setupPlugins(defaultPath); err != nil {
107 + return nil, err
108 +
109 + }
110 +
111 + return createNode(ctx, defaultPath)
112 +}
113 +
114 +// Spawns a node to be used just for this run (i.e. creates a tmp repo)
115 +func spawnEphemeral(ctx context.Context) (icore.CoreAPI, error) {
116 + if err := setupPlugins(""); err != nil {
117 + return nil, err
118 + }
119 +
120 + // Create a Temporary Repo
121 + repoPath, err := createTempRepo(ctx)
122 + if err != nil {
123 + return nil, fmt.Errorf("failed to create temp repo: %s", err)
124 + }
125 +
126 + // Spawning an ephemeral IPFS node
127 + return createNode(ctx, repoPath)
128 +}
129 +
130 +//
131 +
132 +func connectToPeers(ctx context.Context, ipfs icore.CoreAPI, peers []string) error {
133 + var wg sync.WaitGroup
134 + peerInfos := make(map[peer.ID]*peerstore.PeerInfo, len(peers))
135 + for _, addrStr := range peers {
136 + addr, err := ma.NewMultiaddr(addrStr)
137 + if err != nil {
138 + return err
139 + }
140 + pii, err := peerstore.InfoFromP2pAddr(addr)
141 + if err != nil {
142 + return err
143 + }
144 + pi, ok := peerInfos[pii.ID]
145 + if !ok {
146 + pi = &peerstore.PeerInfo{ID: pii.ID}
147 + peerInfos[pi.ID] = pi
148 + }
149 + pi.Addrs = append(pi.Addrs, pii.Addrs...)
150 + }
151 +
152 + wg.Add(len(peerInfos))
153 + for _, peerInfo := range peerInfos {
154 + go func(peerInfo *peerstore.PeerInfo) {
155 + defer wg.Done()
156 + err := ipfs.Swarm().Connect(ctx, *peerInfo)
157 + if err != nil {
158 + log.Printf("failed to connect to %s: %s", peerInfo.ID, err)
159 + }
160 + }(peerInfo)
161 + }
162 + wg.Wait()
163 + return nil
164 +}
165 +
166 +func getUnixfsFile(path string) (files.File, error) {
167 + file, err := os.Open(path)
168 + if err != nil {
169 + return nil, err
170 + }
171 + defer file.Close()
172 +
173 + st, err := file.Stat()
174 + if err != nil {
175 + return nil, err
176 + }
177 +
178 + f, err := files.NewReaderPathFile(path, file, st)
179 + if err != nil {
180 + return nil, err
181 + }
182 +
183 + return f, nil
184 +}
185 +
186 +func getUnixfsNode(path string) (files.Node, error) {
187 + st, err := os.Stat(path)
188 + if err != nil {
189 + return nil, err
190 + }
191 +
192 + f, err := files.NewSerialFile(path, false, st)
193 + if err != nil {
194 + return nil, err
195 + }
196 +
197 + return f, nil
198 +}
199 +
200 +/// -------
201 +
202 +func main() {
203 + /// --- Part I: Getting a IPFS node running
204 +
205 + fmt.Println("-- Getting an IPFS node running -- ")
206 +
207 + ctx, _ := context.WithCancel(context.Background())
208 +
209 + /*
210 + // Spawn a node using the default path (~/.ipfs), assuming that a repo exists there already
211 + fmt.Println("Spawning node on default repo")
212 + ipfs, err := spawnDefault(ctx)
213 + if err != nil {
214 + fmt.Println("No IPFS repo available on the default path")
215 + }
216 + */
217 +
218 + // Spawn a node using a temporary path, creating a temporary repo for the run
219 + fmt.Println("Spawning node on a temporary repo")
220 + ipfs, err := spawnEphemeral(ctx)
221 + if err != nil {
222 + panic(fmt.Errorf("failed to spawn ephemeral node: %s", err))
223 + }
224 +
225 + fmt.Println("IPFS node is running")
226 +
227 + /// --- Part II: Adding a file and a directory to IPFS
228 +
229 + fmt.Println("\n-- Adding and getting back files & directories --")
230 +
231 + inputBasePath := "./example-folder/"
232 + inputPathFile := inputBasePath + "ipfs.paper.draft3.pdf"
233 + inputPathDirectory := inputBasePath + "test-dir"
234 +
235 + someFile, err := getUnixfsNode(inputPathFile)
236 + if err != nil {
237 + panic(fmt.Errorf("Could not get File: %s", err))
238 + }
239 +
240 + cidFile, err := ipfs.Unixfs().Add(ctx, someFile)
241 + if err != nil {
242 + panic(fmt.Errorf("Could not add File: %s", err))
243 + }
244 +
245 + fmt.Printf("Added file to IPFS with CID %s\n", cidFile.String())
246 +
247 + someDirectory, err := getUnixfsNode(inputPathDirectory)
248 + if err != nil {
249 + panic(fmt.Errorf("Could not get File: %s", err))
250 + }
251 +
252 + cidDirectory, err := ipfs.Unixfs().Add(ctx, someDirectory)
253 + if err != nil {
254 + panic(fmt.Errorf("Could not add Directory: %s", err))
255 + }
256 +
257 + fmt.Printf("Added directory to IPFS with CID %s\n", cidDirectory.String())
258 +
259 + /// --- Part III: Getting the file and directory you added back
260 +
261 + outputBasePath := "./example-folder/"
262 + outputPathFile := outputBasePath + strings.Split(cidFile.String(), "/")[2]
263 + outputPathDirectory := outputBasePath + strings.Split(cidDirectory.String(), "/")[2]
264 +
265 + rootNodeFile, err := ipfs.Unixfs().Get(ctx, cidFile)
266 + if err != nil {
267 + panic(fmt.Errorf("Could not get file with CID: %s", err))
268 + }
269 +
270 + err = files.WriteTo(rootNodeFile, outputPathFile)
271 + if err != nil {
272 + panic(fmt.Errorf("Could not write out the fetched CID: %s", err))
273 + }
274 +
275 + fmt.Printf("Got file back from IPFS (IPFS path: %s) and wrote it to %s\n", cidFile.String(), outputPathFile)
276 +
277 + rootNodeDirectory, err := ipfs.Unixfs().Get(ctx, cidDirectory)
278 + if err != nil {
279 + panic(fmt.Errorf("Could not get file with CID: %s", err))
280 + }
281 +
282 + err = files.WriteTo(rootNodeDirectory, outputPathDirectory)
283 + if err != nil {
284 + panic(fmt.Errorf("Could not write out the fetched CID: %s", err))
285 + }
286 +
287 + fmt.Printf("Got directory back from IPFS (IPFS path: %s) and wrote it to %s\n", cidDirectory.String(), outputPathDirectory)
288 +
289 + /// --- Part IV: Getting a file from the IPFS Network
290 +
291 + fmt.Println("\n-- Going to connect to a few nodes in the Network as bootstrappers --")
292 +
293 + bootstrapNodes := []string{
294 + // IPFS Bootstrapper nodes.
295 + "/dnsaddr/bootstrap.libp2p.io/ipfs/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
296 + "/dnsaddr/bootstrap.libp2p.io/ipfs/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
297 + "/dnsaddr/bootstrap.libp2p.io/ipfs/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
298 + "/dnsaddr/bootstrap.libp2p.io/ipfs/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt",
299 +
300 + // IPFS Cluster Pinning nodes
301 + "/ip4/138.201.67.219/tcp/4001/ipfs/QmUd6zHcbkbcs7SMxwLs48qZVX3vpcM8errYS7xEczwRMA",
302 + "/ip4/138.201.67.220/tcp/4001/ipfs/QmNSYxZAiJHeLdkBg38roksAR9So7Y5eojks1yjEcUtZ7i",
303 + "/ip4/138.201.68.74/tcp/4001/ipfs/QmdnXwLrC8p1ueiq2Qya8joNvk3TVVDAut7PrikmZwubtR",
304 + "/ip4/94.130.135.167/tcp/4001/ipfs/QmUEMvxS2e7iDrereVYc5SWPauXPyNwxcy9BXZrC1QTcHE",
305 +
306 + // You can add more nodes here, for example, another IPFS node you might have running locally, mine was:
307 + // "/ip4/127.0.0.1/tcp/4010/ipfs/QmZp2fhDLxjYue2RiUvLwT9MWdnbDxam32qYFnGmxZDh5L",
308 + }
309 +
310 + go connectToPeers(ctx, ipfs, bootstrapNodes)
311 +
312 + exampleCIDStr := "QmUaoioqU7bxezBQZkUcgcSyokatMY71sxsALxQmRRrHrj"
313 +
314 + fmt.Printf("Fetching a file from the network with CID %s\n", exampleCIDStr)
315 + outputPath := outputBasePath + exampleCIDStr
316 + testCID := icorepath.New(exampleCIDStr)
317 +
318 + rootNode, err := ipfs.Unixfs().Get(ctx, testCID)
319 + if err != nil {
320 + panic(fmt.Errorf("Could not get file with CID: %s", err))
321 + }
322 +
323 + err = files.WriteTo(rootNode, outputPath)
324 + if err != nil {
325 + panic(fmt.Errorf("Could not write out the fetched CID: %s", err))
326 + }
327 +
328 + fmt.Printf("Wrote the file to %s\n", outputPath)
329 +
330 + fmt.Println("\nAll done! You just finalized your first tutorial on how to use go-ipfs as a library")
331 +}