809fc0569 · cryptotaxi247/kubo · siGit Code & Deploy

@@ -111,7 +111,8 @@ installing go-ipfs and visiting http://localhost:5001/webui.

111 ### Performance

112

113 This release includes some significant performance improvements, both in terms

114 -of resource utilization and speed.

114 +of resource utilization and speed. This section will go into some technical

115 +details so feel free to skip it if you're just looking for shiny new features.

116

117 #### Resource Utilization

118

@@ -119,26 +120,48 @@ In this release, we've (a) fixed a slow memory leak in libp2p and (b)

120 significantly reduced the allocation load. Together, these should improve both

121 memory and CPU usage.

122

122 -... TODO ...

123 +##### Datastructures

124

124 -* CIDs

125 -* Multiaddrs

126 -* Peerstore revert

127 -* Yamux buffering

128 -* ...

125 +We've changed two of our most frequently used datastructures, CIDs and

126 +Multiaddrs, to reduce allocation load.

127

130 -#### Bitswap Performance

128 +First, we now store CIDs *encode* as strings, instead of decoded in structs

129 +(behind pointers). In addition to being more compact, our `Cid` type is now a

130 +valid `map` key so we no longer have to encode CIDs every time we want to use

131 +them in a map/set. Allocations when inserting CIDs into maps/sets was showing up

132 +as a significant source of allocations under heavy load so this change should

133 +improve memory usage.

134 +

135 +Second, we've changed many of our multiaddr parsing/processing/formatting

136 +functions to allocate less. Much of our DHT related-work includes processing

137 +multiaddrs so this should reduce CPU utilization when heavily using the DHT.

138 +

139 +##### Streams and Yamux

140 +

141 +Streams have always plagued us in terms of memory utilization. This was

142 +partially solved by introducing the connection manager, keeping our maximum

143 +connection count to a reasonable number but they're still a major memory sink.

144 +

145 +This release sees two improvements on this front:

146

132 -TODO: Multiple blocks per message. Could someone test this? Basic tests showed

133 -some improvements for small files but I'd like to see some numbers.

147 +1. A memory [leak in identify](https://github.com/libp2p/go-libp2p/issues/419)

148 + has been fixed. This was slowly causing us to leak connections (locking up

149 + the memory used by the connections' streams).

150 +2. Yamux streams now use a buffer-pool backed, auto shrinking read buffer.

151 + Before, this read buffer would grow to its maximum size (a few megabytes) and

152 + never shrink but these buffers now shrink as they're emptied.

153

135 -#### Async Directory Listing V1

154 +#### Bitswap Performance

155

137 -(v2 will make it into the next release)

156 +Bitswap will now pack *multiple* small blocks into a single message thanks

157 +[ipfs/go-bitswap#5](https://github.com/ipfs/go-bitswap/pull/5). While this won't

158 +help when transferring large files (with large blocks), this should help when

159 +transferring many tiny files.

160

139 -TODO

161 +### Refactors and Endeavors

162

141 -### Efforts

163 +This release saw yet another commands-library refactor, work towards the

164 +CoreAPI, and the first step towards reliable base32 CID support.

165

166 #### Commands Lib

167

@@ -160,25 +183,49 @@ interface using this API, which could help performance in some use cases.

183

184 You can track progress in https://github.com/ipfs/go-ipfs/issues/4498

185

163 -

186 #### CIDv1/Base32 Migration

187

166 -FIXME: Someone in charge of the DWEB effort should review and edit this section.

167 -

168 -In order to support CIDs in dweb links (CID.ipfs.dweb.link) this

169 -release contains two important changes:

170 -

171 -(1) The previous mentioned `ipfs cid base32` command for converting

172 -CID to a case intensive encoding required by domain names. This

173 -command converts a CID to version 1 and encodes it using base32.

174 -

175 -(2) Unfortunately when a CID is converted to base32 the version of the

176 -CID also changes which changes the binary representation of the CID.

177 -This means that up until this release trying to retrieve CID version 0

178 -content using CID version 1 will fail. This release adds a hack so

179 -that so that content is available regardless of the CID version used

180 -making retrieving content via a CIDv1/Base32 link more reliable once

181 -enough nodes are upgraded.

188 +Currently, IPFS is usually used in browsers by browsing to

189 +`https://SOME_GATEWAY/ipfs/CID/...`. There are two significant drawbacks to this

190 +approach:

191 +

192 +1. From a browser security standpoint, all IPFS "sites" will live under the same

193 + origin (SOME_GATEWAY).

194 +2. From a UX standpoint, this doesn't feel very "native" (even if the gateway is

195 + a local IPFS node).

196 +

197 +To fix the security issue, we intend to switch IPFS gateway links

198 +`https://ipfs.io/ipfs/CID` to to `https://CID.ipfs.dweb.link`. This way, the CID

199 +will be a part of the

200 +["origin"](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Origin) so

201 +each IPFS website will get a separate security origin.

202 +

203 +To fix the UX issue, we've been working on adding support for `ipfs://CID/...`

204 +to web browsers through our

205 +[ipfs-companion](https://github.com/ipfs/ipfs-companion/) add-on and some new,

206 +experimental extension APIs from Mozilla. This has the same effect of putting

207 +the CID in the URL origin but has the added benefit of looking "native".

208 +

209 +Unfortunately, origins must be *case insensitive*. Currently, most CIDs users

210 +see are *CIDv0* CIDs (those starting with `Qm`) which are *always* base58

211 +encoded and are therefore case-sensitive.

212 +

213 +Fortunately, CIDv1 (the latest CID format) supports arbitrary bases using the

214 +[multibase](https://github.com/multiformats/multibase/) standard. Unfortunately,

215 +IPFS has always treated equivalent CIDv0 and CIDv1 CIDs as distinct. This means

216 +that files added with CIDv0 CIDs (the default) can't be looked up using the

217 +equivalent CIDv1.

218 +

219 +This release makes some significant progress towards solving this issue by

220 +introducing two features:

221 +

222 +(1) The previous mentioned `ipfs cid base32` command for converting CID to a

223 +case intensive encoding required by domain names. This command converts a CID to

224 +version 1 and encodes it using base32.

225 +

226 +(2) A hack to allow locally looking up blocks associated with a CIDv0 CID using

227 +the equivalent CIDv1 CID. This hack will eventually (likely in the next release)

228 +be replaced with a proper CID-version agnostic datastore.

229

230 ### go-ipfs changelog

231

flesh out the rest of the changelog