@cryptotaxi247 / kubo / commits / 31a134283

feat: initial update to the changelog for 0.5.0

Steven Allen committed Mar 10, 2020 at 23:45 UTC 31a13428331974949b0187f4da9ec4f38359ca0d
1 file changed +327
CHANGELOG.md
+327
@@ -1,5 +1,332 @@
1 # go-ipfs changelog
2
3 +## 0.5.0 RC1 2020-04-06
4 +
5 +**WARNING** THIS IS A DRAFT! It highlights some of the new features, along with
6 +a bunch of important errata. But it's _definitely_ missing a _ton_ of shiny new
7 +features.
8 +
9 +### Highlights & Errata
10 +
11 +This release includes many important changes users should be aware of.
12 +
13 +#### New DHT
14 +
15 +This release includes an almost completely rewritten DHT implementation with a
16 +new protocol version. From a user's perspective, providing content, finding
17 +content, and resolving IPNS records should simply get faster. However, this is a
18 +_significant_ (albeit well tested) change and significant changes are always
19 +risky, so heads up.
20 +
21 +##### Old v. New
22 +
23 +The current DHT suffers from three core issues addressed in this release:
24 +
25 +1. Most peers in the DHT cannot be dialed (e.g., due to firewalls and NATs).
26 + Much of a DHT query time is wasted trying to connect to peers that cannot be
27 + reached.
28 +2. The DHT query logic doesn't properly terminate when it hits the end of the
29 + query and, instead, aggressively keeps on searching.
30 +3. The routing tables are poorly maintained. This can cause a search that should
31 + be logarithmic in the size of the network to be linear.
32 +
33 +###### Reachable
34 +
35 +We have addressed the problem of undialable nodes by having nodes wait to join
36 +the DHT as "server" nodes until they've confirmed that they are reachable from
37 +the public internet. Additionally, we've introduced:
38 +
39 +* A new libp2p protocol to push updates to our peers when we start/stop listen
40 + on protocols.
41 +* A libp2p event bus for processing updates like these.
42 +* A new DHT protocol version. New DHT nodes will not admit old DHT nodes into
43 + their routing tables. Old DHT nodes will still be able to issue queries
44 + against the new DHT, they just won't be queried or referred by new DHT nodes.
45 + This way, old, potentially unreachable nodes with bad routing tables won't
46 + pollute the new DHT.
47 +
48 +Unfortunately, there's a significant downside to this approach: VPNs, offline
49 +LANs, etc. where _all_ nodes on the network have private IP addresses and never
50 +communicate over the public internet. In this case, none of these nodes would be
51 +"publicly reachable".
52 +
53 +To address this last point, go-ipfs 0.5.0 will run _two_ DHTs: one for private
54 +networks and one for the public internet. That is, every node will participate
55 +in a LAN DHT and a public WAN DHT.
56 +
57 +**RC1 NOTE:** Several of these features have not been enabled in RC1:
58 +
59 +1. We haven't yet switched the protocol version and will be running the DHT in
60 + "compatibility mode" with the old DHT. Once we flip the switch and enable the
61 + new protocol version, we will need to ensure that at least 20% of the
62 + publicly reachable DHT speaks the new protocol, all at once. The plan is to
63 + introduce a large number of "booster" nodes while the network transitions.
64 +2. We haven't yet introduced the split LAN/WAN DHTs. We're still testing this
65 + approach and considering alternatives.
66 +3. Because we haven't introduced the LAN/WAN DHT split, IPFS nodes running in
67 + DHT server mode will continue to run in DHT server mode _without_ waiting to
68 + confirm that they're reachable from the public internet. Otherwise, we'd
69 + break IPFS nodes running DHTs in VPNs and disconnected LANs.
70 +
71 +###### Query Logic
72 +
73 +We've fixed the DHT query logic by correctly implementing Kademlia (with a few
74 +tweaks). This should significantly speed up:
75 +
76 +* Publishing IPNS & provider records. We previously continued searching for
77 + closer and closer peers to the "target" until we timed out, then we put to the
78 + closest peers we found.
79 +* Resolving IPNS addresses. We previously continued IPNS record searches until
80 + we ran out of peers to query, timed out, or found 16 records.
81 +
82 +In both cases, we now continue till we find the closest peers then stop.
83 +
84 +###### Routing Tables
85 +
86 +Finally, we've addressed the poorly maintained routing tables by:
87 +
88 +* Reducing the likelihood that the connection manager will kill connections to
89 + peers in the routing table.
90 +* Keeping peers in the routing table, even if we get disconnected from them.
91 +* Actively and frequently querying the DHT to keep our routing table full.
92 +
93 +##### Testing
94 +
95 +The DHT rewrite was made possible by our new testing framework,
96 +[testground](https://github.com/ipfs/testground), which allows us to spin up
97 +multi-thousand node tests with simulated real-world network conditions. With
98 +testground and some custom analysis tools, we were able to gain confidence that
99 +the new DHT implementation behaves correctly.
100 +
101 +#### Refactored Bitswap
102 +
103 +This release includes a _major_ [bitswap refactor][bitswap-refactor] running a
104 +new, but backwards compatible, bitswap protocol. We expect these changes to
105 +improve performance significantly.
106 +
107 +With the refactored bitswap, we expect:
108 +
109 +* Few to no duplicate blocks when fetching data from other nodes speaking the
110 + _new_ protocol.
111 +* Better parallelism when fetching from multiple peers.
112 +
113 +Note, the new bitswap won't magically make downloading content any faster until
114 +both seeds and leaches have updated. If you're one of the first to upgrade to
115 +0.5.0 and try downloading from peers that haven't upgraded, you're unlikely to
116 +see much of a performance improvement, if any.
117 +
118 +[bitswap-refactor]: https://blog.ipfs.io/2020-02-14-improved-bitswap-for-container-distribution/
119 +
120 +#### Provider Record Changes
121 +
122 +When you add content to your IPFS node, you advertise this content to the
123 +network by announcing it in the DHT. We call this "providing".
124 +
125 +However, go-ipfs has multiple ways to address the same underlying bytes.
126 +Specifically, we address content by content ID (CID) and the same underlying
127 +bytes can be addressed using (a) two different versions of CIDs (CIDv1 and
128 +CIDv2) and (b) with different "codecs" depending on how we're interpreting the
129 +data.
130 +
131 +Prior to go-ipfs 0.5.0, we used the content id (CID) in the DHT when sending out
132 +provider records for content. Unfortunately, this meant that users trying to
133 +find data announced using one CID wouldn't find nodes providing the content
134 +under a different CID.
135 +
136 +In go-ipfs 0.5.0, we're announcing data by _multihash_, not _CID_. This way,
137 +regardless of the CID version used by the peer adding the content, the peer
138 +trying to download the content should still be able to find it.
139 +
140 +**Warning:** as part of the network, this could impact finding content added
141 +with CIDv1. Because go-ipfs 0.5.0 will announce and search for content using the
142 +bare multihash (equivalent to the v0 CID), go-ipfs 0.5.0 will be unable to find
143 +CIDv1 content published by nodes prior to go-ipfs 0.5.0 and vice-versa. As CIDv1
144 +is _not_ enabled by default so we believe this will have minimal impact.
145 +However, users are _strongly_ encouraged to upgrade as soon as possible.
146 +
147 +#### IPFS/Libp2p Address Format
148 +
149 +If you've ever run a command like `ipfs swarm peers`, you've likely seen paths
150 +that look like `/ip4/193.45.1.24/tcp/4001/ipfs/QmSomePeerID`. These paths are
151 +_not_ file paths, they're multiaddrs; addresses of peers on the network.
152 +
153 +Unfortunately, `/ipfs/Qm...` is _also_ the same path format we use for files.
154 +This release, changes the multiaddr format from
155 +<code>/ip4/193.45.1.24/tcp/4001/<b>ipfs</b>/QmSomePeerID</code> to
156 +<code>/ip4/193.45.1.24/tcp/4001/<b>p2p</b>/QmSomePeerID</code> to make the
157 +distinction clear.
158 +
159 +What this means for users:
160 +
161 +* Old-style multiaddrs will still be accepted as inputs to IPFS.
162 +* If you were using a multiaddr library (go, js, etc.) to name _files_ because
163 + `/ipfs/QmSomePeerID` looks like `/ipfs/QmSomeFile`, your tool may break if you
164 + upgrade this library.
165 +* If you're manually parsing multiaddrs and are searching for the string
166 + `/ipfs/`..., you'll need to search for `/p2p/...`.
167 +
168 +
169 +#### Minimum RSA Key Size
170 +
171 +Previously, IPFS did not enforce a minimum RSA key size. In this release, we've
172 +introduced a minimum 2048 bit RSA key size. IPFS generates 2048 bit RSA keys by
173 +default so this shouldn't be an issue for anyone in practice. However, users who
174 +explicitly chose a smaller key size will not be able to communicate with new
175 +nodes.
176 +
177 +Unfortunately, the some of the bootstrap peers _did_ intentionally generate 1024
178 +bit RSA keys so they'd have vanity peer addresses (starting with QmSoL for
179 +"solar net"). All IPFS nodes should _also_ have peers with >= 2048 bit RSA keys
180 +in their bootstrap list, but we've introduced a migration to ensure this.
181 +
182 +We implemented this change to follow security best practices and to remove a
183 +potential foot-gun. However, in practice, the security impact of allowing
184 +insecure RSA keys should have been next to none because IPFS doesn't trust other
185 +peers on the network anyways.
186 +
187 +#### Subdomain Gateway
188 +
189 +The gateway will redirect from `http://localhost:5001/ipfs/CID/...` to
190 +`http://CID.ipfs.localhost:5001/...` by default. This will:
191 +
192 +* Ensure that every dapp gets its own browser origin.
193 +* Make it easier to write websites that "just work" with IPFS because absolute
194 + paths will now work.
195 +
196 +Paths addressing the gateway by IP address (`http://127.0.0.1:5001/ipfs/CID`)
197 +will not be altered as IP addresses can't have subdomains.
198 +
199 +Note: cURL doesn't follow redirects by default. To avoid breaking cURL and other
200 +clients that don't support redirects, go-ipfs will return the requested file
201 +along with the redirect. Browsers will follow the redirect and abort the
202 +download while cURL will ignore the redirect and finish the download.
203 +
204 +#### TLS By Default
205 +
206 +In this release, we're switching TLS to be the _default_ transport. This means
207 +we'll try to encrypt the connection with TLS before re-trying with SECIO.
208 +
209 +Contrary to the announcement in the go-ipfs 0.4.23 release notes, this release
210 +does not remove SECIO support to maintain compatibility with js-ipfs.
211 +
212 +#### SECIO Deprecation Notice
213 +
214 +SECIO should be considered to be well on the way to deprecation and will be
215 +completely disabled in either the next release (0.6.0, ~mid May) or the one
216 +following that (0.7.0, ~end of June). Before SECIO is disabled, support will be
217 +added for the NOISE transport for compatibility with other IPFS implementations.
218 +
219 +#### QUIC Upgrade
220 +
221 +If you've been using the experimental QUIC support, this release includes
222 +
223 +**RC1 NOTE:** We've temporarily backed out of the new QUIC version because it
224 +currently requires go 1.14 and go 1.14 has some scheduler bugs that go-ipfs can
225 +reliably trigger.
226 +
227 +#### Badger Datastore
228 +
229 +In this release, we're calling the badger datastore (enabled at initialization
230 +with `ipfs init --profile=badgerds`) as stable. However, we're not yet enabling
231 +it by default.
232 +
233 +The benefit of badger is that adding/fetching data to/from badger is
234 +_significantly_ faster than adding/fetching data to/from the default datastore,
235 +flatfs. In some tests, adding data to badger is 32x faster than flatfs (in this
236 +release).
237 +
238 +However,
239 +
240 +1. Badger is complicated while flatfs pushes all the complexity down into the
241 + filesystem itself. That means that flatfs is only likely to loose your data
242 + if your underlying filesystem gets corrupted while there are more
243 + opportunities for badger itself to get corrupted.
244 +2. Badger can use a lot of memory. In this release, we've tuned badger to use
245 + very little (~20MiB) of memory by default. However, it can still produce
246 + large (1GiB) spikes in memory usage when garbage collecting.
247 +3. Badger isn't very aggressive when it comes to garbage collection and we're
248 + still investigating ways to get it to more aggressively clean up after
249 + itself.
250 +
251 +TL;DR: Use badger if performance is your main requirement, you rarely/never
252 +delete anything, and you have some memory to spare.
253 +
254 +#### Systemd Support
255 +
256 +For Linux users, this release includes support for two systemd features: socket
257 +activation and startup/shutdown notifications. This makes it possible to:
258 +
259 +* Start IPFS on demand on first use.
260 +* Wait for IPFS to finish starting before starting services that depend on it.
261 +
262 +You can find the new systemd units in the go-ipfs repo under misc/systemd.
263 +
264 +#### IPFS API Over Unix Domain Sockets
265 +
266 +This release supports exposing the IPFS API over a unix domain socket in the
267 +filesystem. You use this feature, run:
268 +
269 +```bash
270 +> ipfs config Addresses.API "/unix/path/to/socket/location"
271 +```
272 +
273 +#### Repo Migration
274 +
275 +IPFS uses repo migrations to make structural changes to the "repo" (the config,
276 +data storage, etc.) on upgrade.
277 +
278 +This release includes two very simple repo migrations: a config migration to
279 +ensure that the config contains working bootstrap nodes and a keystore migration
280 +to base32 encode all key filenames.
281 +
282 +In general, migrations should not require significant manual intervention.
283 +However, you should be aware of migrations and plan for them.
284 +
285 +* If you update go-ipfs with `ipfs update`, `ipfs update` will run the migration
286 + for you.
287 +* If you start the ipfs daemon with `ipfs daemon --migrate`, ipfs will migrate
288 + your repo for you on start.
289 +
290 +Otherwise, if you want more control over the repo migration process, you can
291 +manually install and run the [repo migration
292 +tool](http://dist.ipfs.io/#fs-repo-migrations).
293 +
294 +#### Bootstrap Peer Changes
295 +
296 +**AUTOMATIC MIGRATION REQUIRED**
297 +
298 +The first migration will update the bootstrap peer list to:
299 +
300 +1. Replace the old bootstrap nodes (ones with peer IDs starting with QmSoL),
301 + with new bootstrap nodes (ones with addresses that start with
302 + `/dnsaddr/bootstrap.libp2p.io`.
303 +2. Rewrite the address format from `/ipfs/QmPeerID` to `/p2p/QmPeerID`.
304 +
305 +We're migrating addresses for a few reasons:
306 +
307 +1. We're using DNS to address the new bootstrap nodes so we can change the
308 + underlying IP addresses as necessary.
309 +2. The new bootstrap nodes use 2048 bit keys while the old bootstrap nodes use
310 + 1024 bit keys.
311 +3. We're normalizing the address format to `/p2p/Qm...`.
312 +
313 +Note: This migration won't _add_ the new bootstrap peers to your config if
314 +you've explicitly removed the old bootstrap peers. It will also leave custom
315 +entries in the list alone. In other words, if you've customized your bootstrap
316 +list, this migration won't clobber your changes.
317 +
318 +#### Keystore Changes
319 +
320 +**AUTOMATIC MIGRATION REQUIRED**
321 +
322 +Go-IPFS stores additional keys (i.e., all keys other than the "identity" key) in
323 +the keystore. You can list these keys with `ipfs key`.
324 +
325 +Currently, the keystore stores keys as regular files, named after the key
326 +itself. Unfortunately, filename restrictions and case-insensitivity are platform
327 +specific. To avoid platform specific issues, we're base32 encoding all key names
328 +and renaming all keys on-disk.
329 +
330 ## 0.4.23 2020-01-29
331
332 Given the large number of fixes merged since 0.4.22, we've decided to cut another patch release.