Doc updates/additions
Steve Loeppky committed
Mar 1, 2023 at 15:12 UTC
f84ad6d627fcd6d70f173229db22084837526ccf
3 files changed
+64
-134
docs/changelogs/v0.19.md
+11
-2
@@ -1,11 +1,12 @@
1
+<!-- omit in toc -->
2
# Kubo changelog v0.19
3
3
-- [v0.19.0](#v0190)
4
-
4
+<!-- omit in toc -->
5
## v0.19.0
6
7
- [Overview](#overview)
8
- [🔦 Highlights](#-highlights)
9
+ - [Improving the libp2p resource management integration](#improving-the-libp2p-resource-management-integration)
10
- [📝 Changelog](#-changelog)
11
- [👨👩👧👦 Contributors](#-contributors)
12
@@ -13,6 +14,14 @@
14
15
### 🔦 Highlights
16
17
+#### Improving the libp2p resource management integration
18
+There are further followups up on libp2p resource manager improvements in Kubo [0.18.0](https://github.com/ipfs/kubo/blob/master/docs/changelogs/v0.18.md#improving-libp2p-resource-management-integration-1)
19
+and [0.18.1](https://github.com/ipfs/kubo/blob/master/docs/changelogs/v0.18.md#improving-libp2p-resource-management-integration):
20
+1. `ipfs swarm limits` and `ipfs swarm stats` have been replaced by `ipfs swarm resources` to provide a single/combined view for limits and their current usage in a more intuitive ordering.
21
+1. Removal of `Swarm.ResourceMgr.Limits` config. Instead [the power user can specify limits in a .json file that are fed directly to go-libp2p](https://github.com/ipfs/kubo/blob/master/docs/libp2p-resource-management.md#user-supplied-override-limits). This allows the power user to take advantage of the [new resource manager types introduced in go-libp2p 0.25](https://github.com/libp2p/go-libp2p/blob/master/CHANGELOG.md#new-resource-manager-types-) including "use default", "unlimited", "block all".
22
+ - Note: we don't expect most users to need these capablities, but they are there if so.
23
+1. [Doc updates](https://github.com/ipfs/kubo/blob/master/docs/libp2p-resource-management.md).
24
+
25
### 📝 Changelog
26
27
### 👨👩👧👦 Contributors
docs/config.md
+2
-60
@@ -146,7 +146,6 @@ config file at runtime.
146
- [`Swarm.ResourceMgr.Enabled`](#swarmresourcemgrenabled)
147
- [`Swarm.ResourceMgr.MaxMemory`](#swarmresourcemgrmaxmemory)
148
- [`Swarm.ResourceMgr.MaxFileDescriptors`](#swarmresourcemgrmaxfiledescriptors)
149
- - [`Swarm.ResourceMgr.Limits`](#swarmresourcemgrlimits)
149
- [`Swarm.ResourceMgr.Allowlist`](#swarmresourcemgrallowlist)
150
- [`Swarm.Transports`](#swarmtransports)
151
- [`Swarm.Transports.Network`](#swarmtransportsnetwork)
@@ -1846,6 +1845,8 @@ This value is also used to scale the limit on various resources at various scope
1845
when the default limits (discussed in [libp2p resource management](./libp2p-resource-management.md)) are used.
1846
For example, increasing this value will increase the default limit for incoming connections.
1847
1848
+It is possible to inspect the runtime limits via `ipfs swarm resources --help`.
1849
+
1850
Default: `[TOTAL_SYSTEM_MEMORY]/2`
1851
Type: `optionalBytes`
1852
@@ -1859,65 +1860,6 @@ This param is ignored on Windows.
1860
Default `[TOTAL_SYSTEM_FILE_DESCRIPTORS]/2`
1861
Type: `optionalInteger`
1862
1862
-#### `Swarm.ResourceMgr.Limits`
1863
-
1864
-Map of resource limits [per scope](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#resource-scopes).
1865
-
1866
-The map supports fields from the [`LimitConfig` struct](https://github.com/libp2p/go-libp2p/blob/master/p2p/host/resource-manager/limit_defaults.go#L111).
1867
-
1868
-[`BaseLimit`s](https://github.com/libp2p/go-libp2p/blob/master/p2p/host/resource-manager/limit.go#L89) can be set for any scope, and within the `BaseLimit`, all limit <key,value>s are optional.
1869
-
1870
-The `Swarm.ResourceMgr.Limits` override the default limits described above.
1871
-Any override `BaseLimits` or limit <key,value>s from `Swarm.ResourceMgr.Limits`
1872
-that aren't specified will use the [computed default limits](./libp2p-resource-management.md#computed-default-limits).
1873
-
1874
-Until [ipfs/kubo#9564](https://github.com/ipfs/kubo/issues/9564) is addressed, there isn't a way to set an override limit of zero.
1875
-0 is currently ignored. 0 currently means use to use the [computed default limits](./libp2p-resource-management.md#computed-default-limits).
1876
-
1877
-Example #1: setting limits for a specific scope
1878
-```json
1879
-{
1880
- "Swarm": {
1881
- "ResourceMgr": {
1882
- "Limits": {
1883
- "System": {
1884
- "Memory": 1073741824,
1885
- "FD": 512,
1886
- "Conns": 1024,
1887
- "ConnsInbound": 256,
1888
- "ConnsOutbound": 1024,
1889
- "Streams": 16384,
1890
- "StreamsInbound": 4096,
1891
- "StreamsOutbound": 16384
1892
- }
1893
- }
1894
- }
1895
- }
1896
-}
1897
-```
1898
-
1899
-Example #2: setting a specific <key,value> limit
1900
-```json
1901
-{
1902
- "Swarm": {
1903
- "ResourceMgr": {
1904
- "Limits": {
1905
- "Transient": {
1906
- "ConnsOutbound": 256,
1907
- }
1908
- }
1909
- }
1910
- }
1911
-}
1912
-```
1913
-
1914
-It is also possible to inspect and adjust some runtime limits via `ipfs swarm stats --help` and `ipfs swarm limit --help`.
1915
-Changes made via `ipfs swarm limit` are persisted in `Swarm.ResourceMgr.Limits`.
1916
-
1917
-Default: `{}` (use the [computed defaults](./libp2p-resource-management.md#computed-default-limits))
1918
-
1919
-Type: `object[string->object]`
1920
-
1863
#### `Swarm.ResourceMgr.Allowlist`
1864
1865
A list of multiaddrs that can bypass normal system limits (but are still limited by the allowlist scope).
docs/libp2p-resource-management.md
+51
-72
@@ -1,3 +1,4 @@
1
+<!-- omit in toc -->
2
# libp2p Network Resource Manager <small>(`Swarm.ResourceMgr`)</small>
3
4
## Purpose
@@ -12,32 +13,27 @@ Good places to start are:
13
3. Understand [how to inspect and change limits](#user-supplied-override-limits)
14
15
## Table of Contents
15
-<!-- TOC depthfrom:2 -->
16
17
- [Purpose](#purpose)
18
- [🙋 Help! The resource manager is protecting my node but I want to understand more](#-help--the-resource-manager-is-protecting-my-node-but-i-want-to-understand-more)
19
- [Table of Contents](#table-of-contents)
20
- [Levels of Configuration](#levels-of-configuration)
21
- - [Approach](#approach)
22
- - [Computed Default Limits](#computed-default-limits)
23
- - [User Supplied Override Limits](#user-supplied-override-limits)
24
- - [Infinite limits](#infinite-limits)
21
+ - [Approach](#approach)
22
+ - [Computed Default Limits](#computed-default-limits)
23
+ - [User Supplied Override Limits](#user-supplied-override-limits)
24
- [FAQ](#faq)
26
- - [What do these "Protected from exceeding resource limits" log messages mean?](#what-do-these-protected-from-exceeding-resource-limits-log-messages-mean)
27
- - [What are the "Application error 0x0 remote ... cannot reserve ..." messages?](#what-are-the-application-error-0x0-remote--cannot-reserve--messages)
28
- - [How does the resource manager ResourceMgr relate to the connection manager ConnMgr?](#how-does-the-resource-manager-resourcemgr-relate-to-the-connection-manager-connmgr)
29
- - [How does one see the Active Limits?](#how-does-one-see-the-active-limits)
30
- - [How does one see the Computed Default Limits?](#how-does-one-see-the-computed-default-limits)
31
- - [How does one monitor libp2p resource usage?](#how-does-one-monitor-libp2p-resource-usage)
25
+ - [What do these "Protected from exceeding resource limits" log messages mean?](#what-do-these-protected-from-exceeding-resource-limits-log-messages-mean)
26
+ - [How does one see the Active Limits?](#how-does-one-see-the-active-limits)
27
+ - [How does one see the Computed Default Limits?](#how-does-one-see-the-computed-default-limits)
28
+ - [How does one monitor libp2p resource usage?](#how-does-one-monitor-libp2p-resource-usage)
29
+ - [How does the resource manager (ResourceMgr) relate to the connection manager (ConnMgr)?](#how-does-the-resource-manager-resourcemgr-relate-to-the-connection-manager-connmgr)
30
+ - [What are the "Application error 0x0 (remote) ... cannot reserve ..." messages?](#what-are-the-application-error-0x0-remote--cannot-reserve--messages)
31
- [History](#history)
32
34
-<!-- /TOC -->
35
-
33
## Levels of Configuration
34
35
See also the [`Swarm.ResourceMgr` config docs](./config.md#swarmresourcemgr).
36
40
-
37
### Approach
38
libp2p's resource manager provides tremendous flexibility but also adds complexity. There are these levels of limit configuration for resource management protection:
39
@@ -45,12 +41,11 @@ libp2p's resource manager provides tremendous flexibility but also adds complexi
41
based on the amount of memory and file descriptors their system has.
42
This should protect the node from many attacks.
43
48
-1. "Slightly more advanced user" - They can tweak the default limits discussed below.
49
- Where the defaults aren't good enough, a good set of higher-level "knobs" are exposed to satisfy most use cases
44
+2. "Slightly more advanced user" - Where the defaults aren't good enough, a good set of higher-level "knobs" are exposed to satisfy most use cases
45
without requiring users to wade into all the intricacies of libp2p's resource manager.
46
The "knobs"/inputs are `Swarm.ResourceMgr.MaxMemory` and `Swarm.ResourceMgr.MaxFileDescriptors` as described below.
47
53
-1. "Power user" - They specify overrides to computed default limits via `ipfs swarm limit` and `Swarm.ResourceMgr.Limits`;
48
+3. "Power user" - They [specify override limits](#user-supplied-override-limits) and own their own destiny without Kubo getting in the way.
49
50
### Computed Default Limits
51
With the `Swarm.ResourceMgr.MaxMemory` and `Swarm.ResourceMgr.MaxFileDescriptors` inputs defined,
@@ -58,7 +53,7 @@ With the `Swarm.ResourceMgr.MaxMemory` and `Swarm.ResourceMgr.MaxFileDescriptors
53
[system](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#the-system-scope),
54
[transient](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#the-transient-scope),
55
and [peer](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#peer-scopes) scopes.
61
-Other scopes are ignored (by being set to "[~infinity](#infinite-limits])".
56
+Other scopes are ignored (by being set to "unlimited").
57
58
The reason these scopes are chosen is because:
59
- `system` - This gives us the coarse-grained control we want so we can reason about the system as a whole.
@@ -72,13 +67,14 @@ The reason these scopes are chosen is because:
67
(e.g., bug in a peer which is causing it to "misbehave").
68
In the unintentional case, we want to make sure a "misbehaving" node doesn't consume more resources than necessary.
69
75
-Within these scopes, limits are just set on
76
-[memory](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#memory),
77
-[file descriptors (FD)](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#file-descriptors), and [*inbound* connections](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#connections).
70
+Within these scopes, limits are set on:
71
+1. [memory](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#memory)
72
+2. [file descriptors (FD)](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#file-descriptors)
73
+3. [*inbound* connections](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#connections).
74
Limits are set based on the `Swarm.ResourceMgr.MaxMemory` and `Swarm.ResourceMgr.MaxFileDescriptors` inputs above.
75
76
There are also some special cases where minimum values are enforced.
81
-For example, Kubo maintainers have found in practice that it's a footgun to have too low of a value for `Swarm.ResourceMgr.Limits.System.ConnsInbound` and a default minimum is used. (See [core/node/libp2p/rcmgr_defaults.go](https://github.com/ipfs/kubo/blob/master/core/node/libp2p/rcmgr_defaults.go) for specifics.)
77
+For example, Kubo maintainers have found in practice that it's a footgun to have too low of a value for `System.ConnsInbound` and a default minimum is used. (See [core/node/libp2p/rcmgr_defaults.go](https://github.com/ipfs/kubo/blob/master/core/node/libp2p/rcmgr_defaults.go) for specifics.)
78
79
We trust this node to behave properly and thus don't limit *outbound* connection/stream limits.
80
We apply any limits that libp2p has for its protocols/services
@@ -87,29 +83,10 @@ since we assume libp2p knows best here.
83
Source: [core/node/libp2p/rcmgr_defaults.go](https://github.com/ipfs/kubo/blob/master/core/node/libp2p/rcmgr_defaults.go)
84
85
### User Supplied Override Limits
90
-Once Kubo has the [Computed Default Limits](#computed-default-limits), it then applies any user-supplied [`Swarm.ResourceMgr.Limits`](https://github.com/ipfs/kubo/blob/master/docs/config.md#swarmresourcemgrlimits) on top.
91
-These become the [active limits](#how-does-one-see-the-active-limits).
92
-
93
-While `Swarm.ResourceMgr.Limits` can be edited directly, it is also possible to use `ipfs swarm limit` command to inspect and tweak specific limits at runtime.
94
-
95
-To see all resources that are close to hitting their respective limit:
96
-
97
-```console
98
-$ ipfs swarm stats --min-used-limit-perc=90 all
99
-```
100
-
101
-To modify limits for specific scope (e.g. `system`):
102
-
103
-```console
104
-$ ipfs swarm limit system > change.json
105
-$ vi change.json
106
-$ ipfs swarm limit system change.json
107
-```
108
-
109
-Learn more: `ipfs swarm limit --help`
110
-
111
-### Infinite limits
112
-There isn't a way via config to specify infinite limits (see [go-libp2p#1935](https://github.com/libp2p/go-libp2p/issues/1935)). For example, "-1" is not infinity. To work around this, Kubo uses a magic number of "1000000000" to denote infinity since it's effectively infinite.
86
+A user who wants fine control over the limits used by the go-libp2p resoure manager can specify overrides to the [computed default limits](#computed-default-limits).
87
+This is done by defining limits in ``$IPFS_PATH/libp2p-resource-limit-overrides.json``.
88
+These values trump anything else and are parsed directly by go-libp2p.
89
+(See the [go-libp2p Resource Manager README](https://github.com/libp2p/go-libp2p/blob/master/p2p/host/resource-manager/README.md) for formatting.)
90
91
## FAQ
92
@@ -121,58 +98,60 @@ As an example:
98
> Protected from exceeding resource limits 2 times: "system: cannot reserve inbound connection: resource limit exceeded"
99
100
This means that there were 2 recent occurrences where the libp2p resource manager prevented an inbound connection at the "system" [scope](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#resource-scopes).
124
-Specificaly the ``Swarm.ResourceMgr.Limits.System.ConnsInbound`` [active limit](#how-does-one-see-the-active-limits) was hit.
101
+Specificaly the ``System.ConnsInbound`` limit was hit.
102
126
-This can be analyzed by viewing the limit with `ipfs swarm limit system` and comparing the usage with `ipfs swarm stats system`.
127
-`ConnsInbound` is likely close or at the limit value.
103
+This can be analyzed by viewing the limit and current usage with `ipfs swarm resources`.
104
+`System.ConnsInbound` is likely close or at the limit value.
105
129
-The simplest way to identify all resources across all scopes that are close to exceeding their limit is with a command like `ipfs swarm stats --min-used-limit-perc=90 all`.
106
+The simplest way to identify all resources across all scopes that are close to exceeding their limit (>90% usage) is with a command like `ipfs swarm resources | egrep "9.\..%"`
107
108
Sources:
109
* [kubo resource manager logging](https://github.com/ipfs/kubo/blob/master/core/node/libp2p/rcmgr_logging.go)
110
* [libp2p resource manager messages](https://github.com/libp2p/go-libp2p/blob/master/p2p/host/resource-manager/scope.go)
111
135
-### What are the "Application error 0x0 (remote) ... cannot reserve ..." messages?
136
-These are messages coming from a *remote* go-libp2p peer (likely another Kubo node) with the resource manager enabled on why it failed to establish a connection.
112
+### How does one see the Active Limits?
113
+A dump of what limits are actually being used by the resource manager ([Computed Default Limits](#computed-default-limits) + [User Supplied Override Limits](#user-supplied-override-limits))
114
+can be obtained by `ipfs swarm resources`.
115
138
-This can be confusing, but these `Application error 0x0 (remote) ... cannot reserve ...` messages can occur even if your local node has the resource manager disabled.
116
+### How does one see the Computed Default Limits?
117
+This can be observed [seeing the active limits](#how-does-one-see-the-active-limits) assuming one hasn't detoured into "power user" mode with [User Supplied Override Limits](#user-supplied-override-limits).
118
140
-You can distinguish resource manager messages originating from your local node if they're from the `resourcemanager` / `libp2p/rcmgr_logging.go` logger
141
-or you see the string that is unique to Kubo (and not in go-libp2p): "Protected from exceeding resource limits".
119
+### How does one monitor libp2p resource usage?
120
+
121
+For [monitoring libp2p resource usage](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#monitoring),
122
+various `*rcmgr_*` metrics can be accessed as the Prometheus endpoint at `{Addresses.API}/debug/metrics/prometheus` (default: `http://127.0.0.1:5001/debug/metrics/prometheus`).
123
+There are also [pre-built Grafana dashboards](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager/obs/grafana-dashboards) that can be added to a Grafana instance.
124
143
-There is a go-libp2p issue ([#1928](https://github.com/libp2p/go-libp2p/issues/1928)) to make it even clearer that this is an error message originating from a remote peer.
125
+A textual view of current resource usage and a list of services, protocols, and peers can be
126
+obtained via `ipfs swarm stats --help`
127
128
### How does the resource manager (ResourceMgr) relate to the connection manager (ConnMgr)?
129
As discussed [here](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#connmanager-vs-resource-manager)
130
these are separate systems in go-libp2p.
131
Kubo performs sanity checks to ensure that some of the hard limits of the ResourceMgr are sufficiently greater than the soft limits of the ConnMgr.
132
150
-The soft limit of `Swarm.ConnMgr.HighWater` needs to be less than the hard limit `Swarm.ResourceMgr.Limits.System.ConnsInbound` for the configuration to make sense.
133
+The soft limit of `Swarm.ConnMgr.HighWater` needs to be less than the resource manager hard limit `System.ConnsInbound` for the configuration to make sense.
134
This ensures the ConnMgr cleans up connections based on connection priorities before the hard limits of the ResourceMgr are applied.
152
-If `Swarm.ConnMgr.HighWater` is greater than `Swarm.ResourceMgr.Limits.System.ConnsInbound`,
135
+If `Swarm.ConnMgr.HighWater` is greater than resource manager's `System.ConnsInbound`,
136
existing low priority idle connections can prevent new high priority connections from being established.
137
The ResourceMgr doesn't know that the new connection is high priority and simply blocks it because of the limit its enforcing.
138
139
To ensure the ConnMgr and ResourceMgr are congruent, the ResourceMgr [computed default limts](#computed-default-limits) are adjusted such that:
157
-1. `Swarm.ResourceMgr.Limits.System.ConnsInbound` >= `max(Swarm.ConnMgr.HighWater * 2, DefaultResourceMgrMinInboundConns)` AND
158
-2. `Swarm.ResourceMgr.Limits.System.StreamsInbound` is greater than any new/adjusted `Swarm.ResourceMgr.Limits.System.ConnsInbound` value so that there's enough streams per connection.
140
+1. `System.ConnsInbound` >= `max(Swarm.ConnMgr.HighWater * 2, DefaultResourceMgrMinInboundConns)` AND
141
+2. `System.StreamsInbound` is greater than any new/adjusted `Swarm.ResourceMgr.Limits.System.ConnsInbound` value so that there's enough streams per connection.
142
160
-### How does one see the Active Limits?
161
-A dump of what limits are actually being used by the resource manager ([Computed Default Limits](#computed-default-limits) + [User Supplied Override Limits](#user-supplied-override-limits))
162
-can be obtained by `ipfs swarm limit all`.
143
+Source: [core/node/libp2p/rcmgr_defaults.go](https://github.com/ipfs/kubo/blob/master/core/node/libp2p/rcmgr_defaults.go)
144
164
-### How does one see the Computed Default Limits?
165
-This can be observed with an empty [`Swarm.ResourceMgr.Limits`](https://github.com/ipfs/kubo/blob/master/docs/config.md#swarmresourcemgrlimits)
166
-and then [seeing the active limits](#how-does-one-see-the-active-limits).
145
+### What are the "Application error 0x0 (remote) ... cannot reserve ..." messages?
146
+These are messages coming from old (pre go-libp2p 0.26) *remote* go-libp2p peers (likely another older Kubo node) with the resource manager enabled on why it failed to establish a connection.
147
168
-### How does one monitor libp2p resource usage?
148
+This can be confusing, but these `Application error 0x0 (remote) ... cannot reserve ...` messages can occur even if your local node has the resource manager disabled.
149
170
-For [monitoring libp2p resource usage](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#monitoring),
171
-various `*rcmgr_*` metrics can be accessed as the Prometheus endpoint at `{Addresses.API}/debug/metrics/prometheus` (default: `http://127.0.0.1:5001/debug/metrics/prometheus`).
172
-There are also [pre-built Grafana dashboards](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager/obs/grafana-dashboards) that can be added to a Grafana instance.
150
+You can distinguish resource manager messages originating from your local node if they're from the `resourcemanager` / `libp2p/rcmgr_logging.go` logger
151
+or you see the string that is unique to Kubo (and not in go-libp2p): "Protected from exceeding resource limits".
152
+
153
+See more info in this go-libp2p issue ([#1928](https://github.com/libp2p/go-libp2p/issues/1928)). go-libp2p 0.26 / Kubo 0.19 onwards this confusing error message was removed.
154
174
-A textual view of current resource usage and a list of services, protocols, and peers can be
175
-obtained via `ipfs swarm stats --help`
155
156
## History
178
-Kubo first [exposed this functionality in Kubo 0.13](./changelogs/v0.13.md#-libp2p-network-resource-manager-swarmresourcemgr), but it was disabled by default. It was then enabled by default in [Kubo 0.17](./changelogs/v0.17.md#libp2p-resource-management-enabled-by-default). Until that point, Kubo was vulnerable to unbound resource usage which could bring down nodes. Introducing limits like this by default after the fact is tricky, which is why there have been changes and improvements afterwards.
157
+Kubo first [exposed this functionality in Kubo 0.13](./changelogs/v0.13.md#-libp2p-network-resource-manager-swarmresourcemgr), but it was disabled by default. It was then enabled by default in [Kubo 0.17](./changelogs/v0.17.md#libp2p-resource-management-enabled-by-default). Until that point, Kubo was vulnerable to unbound resource usage which could bring down nodes. Introducing limits like this by default after the fact is tricky, which is why there have been changes and improvements afterwards. The general trend since 0.17 with (0.18)[./changeloges/v0.18.md#improving-libp2p-resource-management-integration] and 0.19 has been to simplify and provide less options (and footguns!) for users and better documentation.