Added a debug-guidelines doc
Stolen from this comment https://github.com/ipfs/go-ipfs/issues/2594#issuecomment-212506980 License: MIT Signed-off-by: Richard Littauer <richard.littauer@gmail.com>
Richard Littauer committed
Apr 29, 2016 at 14:05 UTC
0e9224f67114f8a61316f2752b4158928faeb39c
1 file changed
+91
debug-guide.md
new
+91
@@ -0,0 +1,91 @@
1
+# General performance debugging guidelines
2
+
3
+This is a document for helping debug go-ipfs. Please add to it if you can!
4
+
5
+### Table of Contents
6
+- [Beginning](#beginning)
7
+- [Analysing the stack dump](#analysing-the-stack-dump)
8
+- [Analyzing the CPU Profile](#analyzing-the-cpu-profile)
9
+- [Other](#other)
10
+
11
+### Beginning
12
+
13
+When you see ipfs doing something (using lots of CPU, memory, or otherwise
14
+being weird), the first thing you want to do is gather all the relevant
15
+profiling information.
16
+
17
+- goroutine dump
18
+ - `curl localhost:5001/debug/pprof/goroutine\?debug=2 > ipfs.stacks`
19
+- 30 second cpu profile
20
+ - `curl localhost:5001/debug/pprof/profile > ipfs.cpuprof`
21
+- heap trace dump
22
+ - `curl localhost:5001/debug/pprof/heap > ipfs.stacks`
23
+- system information
24
+ - `ipfs diag sys > ipfs.sysinfo`
25
+
26
+Bundle all that up and include a copy of the ipfs binary that you are running
27
+(having the exact same binary is important, it contains debug info).
28
+
29
+You can investigate yourself if you feel intrepid:
30
+
31
+### Analysing the stack dump
32
+
33
+The first thing to look for is hung goroutines -- any goroutine thats been stuck
34
+for over a minute will note that in the trace. It looks something like:
35
+
36
+```
37
+goroutine 2306090 [semacquire, 458 minutes]:
38
+sync.runtime_Semacquire(0xc8222fd3e4)
39
+ /home/whyrusleeping/go/src/runtime/sema.go:47 +0x26
40
+sync.(*Mutex).Lock(0xc8222fd3e0)
41
+ /home/whyrusleeping/go/src/sync/mutex.go:83 +0x1c4
42
+gx/ipfs/QmedFDs1WHcv3bcknfo64dw4mT1112yptW1H65Y2Wc7KTV/yamux.(*Session).Close(0xc8222fd340, 0x0, 0x0)
43
+ /home/whyrusleeping/gopkg/src/gx/ipfs/QmedFDs1WHcv3bcknfo64dw4mT1112yptW1H65Y2Wc7KTV/yamux/session.go:205 +0x55
44
+gx/ipfs/QmWSJzRkCMJFHYUQZxKwPX8WA7XipaPtfiwMPARP51ymfn/go-stream-muxer/yamux.(*conn).Close(0xc8222fd340, 0x0, 0x0)
45
+ /home/whyrusleeping/gopkg/src/gx/ipfs/QmWSJzRkCMJFHYUQZxKwPX8WA7XipaPtfiwMPARP51ymfn/go-stream-muxer/yamux/yamux.go:39 +0x2d
46
+gx/ipfs/QmZK81vcgMhpb2t7GNbozk7qzt6Rj4zFqitpvsWT9mduW8/go-peerstream.(*Conn).Close(0xc8257a2000, 0x0, 0x0)
47
+ /home/whyrusleeping/gopkg/src/gx/ipfs/QmZK81vcgMhpb2t7GNbozk7qzt6Rj4zFqitpvsWT9mduW8/go-peerstream/conn.go:156 +0x1f2
48
+created by gx/ipfs/QmZK81vcgMhpb2t7GNbozk7qzt6Rj4zFqitpvsWT9mduW8/go-peerstream.(*Conn).GoClose
49
+ /home/whyrusleeping/gopkg/src/gx/ipfs/QmZK81vcgMhpb2t7GNbozk7qzt6Rj4zFqitpvsWT9mduW8/go-peerstream/conn.go:131 +0xab
50
+```
51
+
52
+At the top, you can see that this goroutine (number 2306090) has been waiting
53
+to acquire a semaphore for 458 minutes. That seems bad. Looking at the rest of
54
+the trace, we see the exact line it's waiting on is line 47 of runtime/sema.go.
55
+That's not particularly helpful, so we move on. Next, we see that call was made
56
+by line 205 of yamux/session.go in the `Close` method of `yamux.Session`. This
57
+one appears to be the issue.
58
+
59
+Given that information, look for another goroutine that might be
60
+holding the semaphore in question in the rest of the stack dump.
61
+(If you need help doing this, ping and we'll stub this out.)
62
+
63
+There are a few different reasons that goroutines can be hung:
64
+- `semacquire` means we're waiting to take a lock or semaphore.
65
+- `select` means that the goroutine is hanging in a select statement and none of the cases are yielding
66
+anything.
67
+- `chan receive` and `chan send` are waiting for a channel to be received from or sent on, respectively.
68
+- `IO wait` generally means that we are waiting on a socket to read or write data, although it *can* mean we are
69
+waiting on a very slow filesystem.
70
+
71
+If you see any of those tags _without_ a `,
72
+X minutes` suffix, that generally means there isn't a problem -- you just caught
73
+that goroutine in the middle of a short wait for something. If the wait time is
74
+over a few minutes, that either means that goroutine doesn't do much, or
75
+something is pretty wrong.
76
+
77
+### Analyzing the CPU Profile
78
+
79
+The go team wrote an [excellent article on profiling go
80
+programs](http://blog.golang.org/profiling-go-programs). If you've already
81
+gathered the above information, you can skip down to where they start talking
82
+about `go tool pprof`. My go-to method of analyzing these is to run the `web`
83
+command, which generates an SVG dotgraph and opens it in your browser. This is
84
+the quickest way to easily point out where the hot spots in the code are.
85
+
86
+### Other
87
+
88
+If you have any questions, or want us to analyze some weird go-ipfs behaviour,
89
+just let us know, and be sure to include all the profiling information
90
+mentioned at the top.
91
+