Doc improvements for rcmgr
This was done taking a fresh look for someone learning about the resource manager being enabled by default. Moved and expanded documentation to the right places for more visibility. Added an initial changelog entry.
Steve Loeppky committed
Nov 16, 2022 at 08:06 UTC
b1a89c93e4e0ff92f855477a5f4d882b96cd2188
4 files changed
+112
-64
core/node/libp2p/rcmgr.go
+8
-4
@@ -52,18 +52,22 @@ func ResourceManager(cfg config.SwarmConfig) interface{} {
52
return nil, opts, fmt.Errorf("opening IPFS_PATH: %w", err)
53
}
54
55
- limits, err := createDefaultLimitConfig(cfg)
55
+ limitConfig, err := createDefaultLimitConfig(cfg)
56
if err != nil {
57
return nil, opts, err
58
}
59
60
+ // The logic for defaults and overriding with specified SwarmConfig.ResourceMgr.Limits
61
+ // is documented in docs/config.md.
62
+ // Any changes here should be reflected there.
63
if cfg.ResourceMgr.Limits != nil {
64
l := *cfg.ResourceMgr.Limits
62
- l.Apply(limits)
63
- limits = l
65
+ // This effectively overrides the computed default LimitConfig with any vlues from cfg.ResourceMgr.Limits
66
+ l.Apply(limitConfig)
67
+ limitConfig = l
68
}
69
66
- limiter := rcmgr.NewFixedLimiter(limits)
70
+ limiter := rcmgr.NewFixedLimiter(limitConfig)
71
72
str, err := rcmgrObs.NewStatsTraceReporter()
73
if err != nil {
core/node/libp2p/rcmgr_defaults.go
+2
-46
@@ -44,51 +44,8 @@ var noLimitIncrease = rcmgr.BaseLimitIncrease{
44
// This file defines implicit limit defaults used when Swarm.ResourceMgr.Enabled
45
46
// createDefaultLimitConfig creates LimitConfig to pass to libp2p's resource manager.
47
-// libp2p's resource manager provides tremendous flexibility but also adds a lot of complexity.
48
-// The intent of the default config here is to provide good defaults,
49
-// and where the defaults aren't good enough,
50
-// to expose a good set of higher-level "knobs" to users to satisfy most use cases
51
-// without requiring users to wade into all the intricacies of libp2p's resource manager.
52
-//
53
-// The inputs one can specify in SwarmConfig are:
54
-// - cfg.ResourceMgr.MaxMemory: This is the max amount of memory in bytes to allow libp2p to use.
55
-// libp2p's resource manager will prevent additional resource creation while this limit is hit.
56
-// If this value isn't specified, 1/8th of the total system memory is used.
57
-// - cfg.ResourceMgr.MaxFileDescriptors: This is the maximum number of file descriptors to allow libp2p to use.
58
-// libp2p's resource manager will prevent additional file descriptor consumption while this limit is hit.
59
-// If this value isn't specified, the maximum between 1/2 of system FD limit and 4096 is used.
60
-// - Swarm.ConnMgr.HighWater: If a connection manager is specified, libp2p's resource manager
61
-// will allow 2x more connections than the HighWater mark
62
-// so the connection manager has "space and time" to close "least useful" connections.
63
-//
64
-// With these inputs defined, limits are created at the system, transient, and peer scopes.
65
-// Other scopes are ignored (by being set to infinity).
66
-// The reason these scopes are chosen is because:
67
-// - system - This gives us the coarse-grained control we want so we can reason about the system as a whole.
68
-// It is the backstop, and allows us to reason about resource consumption more easily
69
-// since don't have think about the interaction of many other scopes.
70
-// - transient - Limiting connections that are in process of being established provides backpressure so not too much work queues up.
71
-// - peer - The peer scope doesn't protect us against intentional DoS attacks.
72
-// It's just as easy for an attacker to send 100 requests/second with 1 peerId vs. 10 requests/second with 10 peers.
73
-// We are reliant on the system scope for protection here in the malicious case.
74
-// The reason for having a peer scope is to protect against unintentional DoS attacks
75
-// (e.g., bug in a peer which is causing it to "misbehave").
76
-// In the unintional case, we want to make sure a "misbehaving" node doesn't consume more resources than necessary.
77
-//
78
-// Within these scopes, limits are just set on memory, FD, and inbound connections/streams.
79
-// Limits are set based on the inputs above.
80
-// We trust this node to behave properly and thus ignore outbound connection/stream limits.
81
-// We apply any limits that libp2p has for its protocols/services
82
-// since we assume libp2p knows best here.
83
-//
84
-// This leaves 3 levels of resource management protection:
85
-// 1. The user who does nothing and uses defaults - In this case they get some sane defaults
86
-// based on the amount of memory and file descriptors their system has.
87
-// This should protect the node from many attacks.
88
-// 2. Slightly more advanced user - They can tweak the above by passing in config on
89
-// maxMemory, maxFD, or maxConns with Swarm.HighWater.ConnMgr.
90
-// 3. Power user - They specify all the limits they want set via Swarm.ResourceMgr.Limits
91
-// and we don't do any defaults/overrides. We pass that config blindly into libp2p resource manager.
47
+// The defaults follow the documentation in docs/config.md.
48
+// Any changes in the logic here should be reflected there.
49
func createDefaultLimitConfig(cfg config.SwarmConfig) (rcmgr.LimitConfig, error) {
50
maxMemoryDefaultString := humanize.Bytes(uint64(memory.TotalMemory()) / 8)
51
maxMemoryString := cfg.ResourceMgr.MaxMemory.WithDefault(maxMemoryDefaultString)
@@ -105,7 +62,6 @@ func createDefaultLimitConfig(cfg config.SwarmConfig) (rcmgr.LimitConfig, error)
62
FD: int(numFD),
63
64
// By default, we just limit connections on the inbound side.
108
- // Note that the limit gets adjusted below if "cfg.ConnMgr.HighWater" is set.
65
Conns: bigEnough,
66
ConnsInbound: rcmgr.DefaultLimits.SystemBaseLimit.ConnsInbound, // same as libp2p default
67
ConnsOutbound: bigEnough,
docs/changelogs/v0.17.md
+22
@@ -20,6 +20,28 @@ Below is an outline of all that is in this release, so you get a sense of all th
20
21
<!-- TODO -->
22
23
+#### libp2p resource management enabled by default
24
+
25
+To help protect nodes from DoS (resource exhaustion) and eclipse attacks,
26
+go-libp2p released a [Network Resource Manager](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manage) with a host of improvements throughout 2022.
27
+
28
+Kubo first [exposed this functionality in Kubo 0.13](https://github.com/ipfs/kubo/blob/master/docs/changelogs/v0.13.md#-libp2p-network-resource-manager-swarmresourcemgr),
29
+but it was disabled by default.
30
+
31
+The resource manager is now enabled by default to protect nodes.
32
+The defaults balance providing protection from various attacks while still enabling normal usecases to work as expected.
33
+
34
+If you want to adjust the defaults, then you can:
35
+1. bound the amount of memory and file descriptors that libp2p will use with [Swarm.ResourceMgr.MaxMemory](https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#swarmresourcemgrmaxmemory)
36
+and Swarm.ResourceMgr.MaxFileDescriptors](https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#swarmresourcemgrmaxfiledescriptors) and/or
37
+2. override any specific resource scopes/limits with [Swarm.ResourceMgr.Limits](https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#swarmresourcemgrlimits)
38
+
39
+See [Swarm.ResourceMgr](https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#swarmresourcemgr) for
40
+1. what limits are set by default,
41
+2. example override configuration,
42
+3. how to access prometheus metrics and view grafana dashboards of resource usage, and
43
+4. how to set explicit "allow lists" to protect against eclipse attacks.
44
+
45
#### Implicit connection manager limits
46
47
Starting with this release, `ipfs init` will no longer store the default
docs/config.md
+80
-14
@@ -1808,30 +1808,76 @@ Type: `optionalDuration`
1808
1809
### `Swarm.ResourceMgr`
1810
1811
-The [libp2p Network Resource Manager](https://github.com/libp2p/go-libp2p-resource-manager#readme) allows setting limits per a scope,
1811
+The [libp2p Netowrk Resource Manager](https://github.com/libp2p/go-libp2p-resource-manager#readme) allows setting limits per [Resource Scope](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#resource-scopes),
1812
and tracking recource usage over time.
1813
1814
-#### `Swarm.ResourceMgr.Enabled`
1814
+** Levels of Configuration **
1815
+libp2p's resource manager provides tremendous flexibility but also adds a lot of complexity.
1816
+There are these levels of limit configuration for resource management protection:
1817
+1. "The user who does nothing" - In this case they get some sane defaults discussed below
1818
+ based on the amount of memory and file descriptors their system has.
1819
+ This should protect the node from many attacks.
1820
+2. "Slightly more advanced user" - They can tweak the default limits discussed below.
1821
+ Where the defaults aren't good enough, a good set of higher-level "knobs" are exposed to satisfy most use cases
1822
+ without requiring users to wade into all the intricacies of libp2p's resource manager.
1823
+ The "knobs"/inputs are `Swarm.ResourceMgr.MaxMemory` and `Swarm.ResourceMgr.MaxFileDescriptors` as described below.
1824
+3. "Power user" - They specify all the default limits from below they want override via `Swarm.ResourceMgr.Limits`;
1825
+
1826
+** Default Limits **
1827
+With these inputs defined, [resource manager limits](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#limits) are created at the
1828
+[system](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#the-system-scope),
1829
+[transient](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#the-transient-scope),
1830
+and [peer](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#peer-scopes) scopes.
1831
+Other scopes are ignored (by being set to "~infinity".
1832
+
1833
+The reason these scopes are chosen is because:
1834
+- system - This gives us the coarse-grained control we want so we can reason about the system as a whole.
1835
+ It is the backstop, and allows us to reason about resource consumption more easily
1836
+ since don't have think about the interaction of many other scopes.
1837
+- transient - Limiting connections that are in process of being established provides backpressure so not too much work queues up.
1838
+- peer - The peer scope doesn't protect us against intentional DoS attacks.
1839
+ It's just as easy for an attacker to send 100 requests/second with 1 peerId vs. 10 requests/second with 10 peers.
1840
+ We are reliant on the system scope for protection here in the malicious case.
1841
+ The reason for having a peer scope is to protect against unintentional DoS attacks
1842
+ (e.g., bug in a peer which is causing it to "misbehave").
1843
+ In the unintional case, we want to make sure a "misbehaving" node doesn't consume more resources than necessary.
1844
+
1845
+Within these scopes, limits are just set on
1846
+[memory](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#memory),
1847
+[file descriptors (FD)](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#file-descriptors), [*inbound* connections](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#connections),
1848
+and [*inbound* streams](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#streams).
1849
+Limits are set based on the inputs above.
1850
+We trust this node to behave properly and thus don't limit *outbound* connection/stream limits.
1851
+We apply any limits that libp2p has for its protocols/services
1852
+since we assume libp2p knows best here.
1853
+
1854
+** libp2p resource monitoring **
1855
+For [monitoring libp2p resource usage](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#monitoring),
1856
+various `*rcmgr_*` metrics can be accessed as the prometheus endpoint at `{Addresses.API}/debug/metrics/prometheus` (default: `http://127.0.0.1:5001/debug/metrics/prometheus`).
1857
+There are also [pre-built Grafana dashboards](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager/obs/grafana-dashboards) that can be added to a Grafana instance.
1858
1816
-Enables the libp2p Network Resource Manager and auguments the default limits
1817
-using user-defined ones in `Swarm.ResourceMgr.Limits` (if present).
1859
+#### `Swarm.ResourceMgr.Enabled`
1860
1819
-Various `*rcmgr_*` metrics can be accessed as the prometheus endpoint at `{Addresses.API}/debug/metrics/prometheus` (default: `http://127.0.0.1:5001/debug/metrics/prometheus`)
1861
+Enables the libp2p Resource Manager using limits based on the defaults and/or other configuration as discussed above.
1862
1863
Default: `true`
1822
-
1864
Type: `flag`
1865
1866
#### `Swarm.ResourceMgr.MaxMemory`
1867
1827
-The maximum amount of memory that the libp2p resource manager will allow.
1868
+This is the max amount of memory to allow libp2p to use.
1869
+libp2p's resource manager will prevent additional resource creation while this limit is reached.
1870
+This value is also used to scale the limit on various resources at various scopes
1871
+when the default limits (discuseed above) are used.
1872
+For example, increasing this value will increase the default limit for incoming connections.
1873
1874
Default: `[TOTAL_SYSTEM_MEMORY]/8`
1875
Type: `optionalBytes`
1876
1877
#### `Swarm.ResourceMgr.MaxFileDescriptors`
1878
1834
-Define the maximum number of file descriptors that libp2p can use.
1879
+This is the maximum number of file descriptors to allow libp2p to use.
1880
+libp2p's resource manager will prevent additional file descriptor consumption while this limit is reached.
1881
1882
This param is ignored on Windows.
1883
@@ -1842,21 +1888,26 @@ Type: `optionalInteger`
1888
1889
Map of resource limits [per scope](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#resource-scopes).
1890
1845
-The map supports fields from [`ScalingLimitConfig`](https://github.com/libp2p/go-libp2p/blob/master/p2p/host/resource-manager/limit_defaults.go#L21-L59)
1846
-struct from [go-libp2p-resource-manager](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#readme).
1891
+The map supports fields from the [`LimitConfig` struct](https://github.com/libp2p/go-libp2p/blob/master/p2p/host/resource-manager/limit_defaults.go#L111).
1892
1893
+[`BaseLimit`s](https://github.com/libp2p/go-libp2p/blob/master/p2p/host/resource-manager/limit.go#L89) can be set for any scope, and within the `BaseLimit`, all limit <key,value>s are optional.
1894
+
1895
+The `Swarm.ResourceMgr.Limits` override the default limits described above.
1896
+Any override `BaseLimits` or limit <key,value>s from `Swarm.ResourceMgr.Limits`
1897
+that aren't specified will use the default limits.
1898
+
1899
+Example #1: setting limits for a specific scope
1900
```json
1901
{
1902
"Swarm": {
1903
"ResourceMgr": {
1852
- "Enabled": true,
1904
"Limits": {
1905
"System": {
1906
+ "Memory": 1073741824,
1907
+ "FD": 512,
1908
"Conns": 1024,
1909
"ConnsInbound": 256,
1910
"ConnsOutbound": 1024,
1858
- "FD": 512,
1859
- "Memory": 1073741824,
1911
"Streams": 16384,
1912
"StreamsInbound": 4096,
1913
"StreamsOutbound": 16384
@@ -1867,13 +1918,28 @@ struct from [go-libp2p-resource-manager](https://github.com/libp2p/go-libp2p/tre
1918
}
1919
```
1920
1921
+Example #2: setting a specific <key,value> limit
1922
+```json
1923
+{
1924
+ "Swarm": {
1925
+ "ResourceMgr": {
1926
+ "Limits": {
1927
+ "Transient": {
1928
+ "ConnsOutbound": 256,
1929
+ }
1930
+ }
1931
+ }
1932
+ }
1933
+}
1934
+```
1935
+
1936
Current resource usage and a list of services, protocols, and peers can be
1937
obtained via `ipfs swarm stats --help`
1938
1939
It is also possible to adjust some runtime limits via `ipfs swarm limit --help`.
1940
Changes made via `ipfs swarm limit` are persisted in `Swarm.ResourceMgr.Limits`.
1941
1876
-Default: `{}` (use the safe implicit defaults)
1942
+Default: `{}` (use the safe implicit defaults described above)
1943
1944
Type: `object[string->object]`
1945