docs(config): improve profile descriptions (#10517)
* docs(config): improve profile descriptions * chore: move profiles and types to the end
Marcin Rataj committed
Sep 20, 2024 at 15:40 UTC
60588afc9eaae23d9e8da3996f2fe7e101b5721d
3 files changed
+224
-196
config/profile.go
+7
-8
@@ -124,7 +124,7 @@ This profile may only be applied when first initializing the node.
124
"flatfs": {
125
Description: `Configures the node to use the flatfs datastore.
126
127
-This is the most battle-tested and reliable datastore.
127
+This is the most battle-tested and reliable datastore.
128
You should use this datastore if:
129
130
* You need a very simple and very reliable datastore, and you trust your
@@ -145,21 +145,20 @@ This profile may only be applied when first initializing the node.
145
},
146
},
147
"badgerds": {
148
- Description: `Configures the node to use the experimental badger datastore.
148
+ Description: `Configures the node to use the legacy badgerv1 datastore.
149
150
-Use this datastore if some aspects of performance,
151
-especially the speed of adding many gigabytes of files, are critical.
152
-However, be aware that:
150
+NOTE: this is badger 1.x, which has known bugs and is no longer supported by the upstream team.
151
+It is provided here only for pre-existing users, allowing them to migrate away to more modern datastore.
152
+
153
+Other caveats:
154
155
* This datastore will not properly reclaim space when your datastore is
156
smaller than several gigabytes. If you run IPFS with --enable-gc, you plan
157
on storing very little data in your IPFS node, and disk usage is more
158
critical than performance, consider using flatfs.
158
-* This datastore uses up to several gigabytes of memory.
159
+* This datastore uses up to several gigabytes of memory.
160
* Good for medium-size datastores, but may run into performance issues
161
if your dataset is bigger than a terabyte.
161
-* The current implementation is based on old badger 1.x
162
- which is no longer supported by the upstream team.
162
163
This profile may only be applied when first initializing the node.`,
164
docs/config.md
+211
-188
@@ -9,16 +9,6 @@ config file at runtime.
9
10
- [The Kubo config file](#the-kubo-config-file)
11
- [Table of Contents](#table-of-contents)
12
- - [Profiles](#profiles)
13
- - [Types](#types)
14
- - [`flag`](#flag)
15
- - [`priority`](#priority)
16
- - [`strings`](#strings)
17
- - [`duration`](#duration)
18
- - [`optionalInteger`](#optionalinteger)
19
- - [`optionalBytes`](#optionalbytes)
20
- - [`optionalString`](#optionalstring)
21
- - [`optionalDuration`](#optionalduration)
12
- [`Addresses`](#addresses)
13
- [`Addresses.API`](#addressesapi)
14
- [`Addresses.Gateway`](#addressesgateway)
@@ -184,184 +174,26 @@ config file at runtime.
174
- [`Version.AgentSuffix`](#versionagentsuffix)
175
- [`Version.SwarmCheckEnabled`](#versionswarmcheckenabled)
176
- [`Version.SwarmCheckPercentThreshold`](#versionswarmcheckpercentthreshold)
187
-
188
-## Profiles
189
-
190
-Configuration profiles allow to tweak configuration quickly. Profiles can be
191
-applied with the `--profile` flag to `ipfs init` or with the `ipfs config profile
192
-apply` command. When a profile is applied a backup of the configuration file
193
-will be created in `$IPFS_PATH`.
194
-
195
-The available configuration profiles are listed below. You can also find them
196
-documented in `ipfs config profile --help`.
197
-
198
-- `server`
199
-
200
- Disables local host discovery, recommended when
201
- running IPFS on machines with public IPv4 addresses.
202
-
203
-- `randomports`
204
-
205
- Use a random port number for the incoming swarm connections.
206
-
207
-- `default-datastore`
208
-
209
- Configures the node to use the default datastore (flatfs).
210
-
211
- Read the "flatfs" profile description for more information on this datastore.
212
-
213
- This profile may only be applied when first initializing the node.
214
-
215
-- `local-discovery`
216
-
217
- Enables local discovery (enabled by default). Useful to re-enable local discovery after it's
218
- disabled by another profile (e.g., the server profile).
219
-
220
-- `test`
221
-
222
- Reduces external interference of IPFS daemon, this
223
- is useful when using the daemon in test environments.
224
-
225
-- `default-networking`
226
-
227
- Restores default network settings.
228
- Inverse profile of the test profile.
229
-
230
-- `flatfs`
231
-
232
- Configures the node to use the flatfs datastore. Flatfs is the default datastore.
233
-
234
- This is the most battle-tested and reliable datastore.
235
- You should use this datastore if:
236
-
237
- - You need a very simple and very reliable datastore, and you trust your
238
- filesystem. This datastore stores each block as a separate file in the
239
- underlying filesystem so it's unlikely to lose data unless there's an issue
240
- with the underlying file system.
241
- - You need to run garbage collection in a way that reclaims free space as soon as possible.
242
- - You want to minimize memory usage.
243
- - You are ok with the default speed of data import, or prefer to use `--nocopy`.
244
-
245
- This profile may only be applied when first initializing the node.
246
-
247
-
248
-- `badgerds`
249
-
250
- Configures the node to use the experimental badger datastore. Keep in mind that this **uses an outdated badger 1.x**.
251
-
252
- Use this datastore if some aspects of performance,
253
- especially the speed of adding many gigabytes of files, are critical. However, be aware that:
254
-
255
- - This datastore will not properly reclaim space when your datastore is
256
- smaller than several gigabytes. If you run IPFS with `--enable-gc`, you plan on storing very little data in
257
- your IPFS node, and disk usage is more critical than performance, consider using
258
- `flatfs`.
259
- - This datastore uses up to several gigabytes of memory.
260
- - Good for medium-size datastores, but may run into performance issues if your dataset is bigger than a terabyte.
261
- - The current implementation is based on old badger 1.x which is no longer supported by the upstream team.
262
-
263
- This profile may only be applied when first initializing the node.
264
-
265
-- `lowpower`
266
-
267
- Reduces daemon overhead on the system. Affects node
268
- functionality - performance of content discovery and data
269
- fetching may be degraded. Local data won't be announced on routing systems like Amino DHT.
270
-
271
- - `Swarm.ConnMgr` set to maintain minimum number of p2p connections at a time.
272
- - Disables [`Reprovider`](#reprovider) service → no CID will be announced on Amino DHT and other routing systems(!)
273
- - Disables AutoNAT.
274
-
275
- Use this profile with caution.
276
-
277
-- `legacy-cid-v0`
278
-
279
- Makes UnixFS import (`ipfs add`) produce legacy CIDv0 with no raw leaves, sha2-256 and 256 KiB chunks.
280
-
281
- > [!WARNING]
282
- > This profile is provided for legacy users and should not be used for new projects.
283
-
284
-- `test-cid-v1`
285
-
286
- Makes UnixFS import (`ipfs add`) produce modern CIDv1 with raw leaves, sha2-256 and 1 MiB chunks.
287
-
288
- > [!NOTE]
289
- > This profile will become the new implicit default, provided for testing purposes.
290
- > Follow [kubo#4143](https://github.com/ipfs/kubo/issues/4143) for more details.
291
-
292
-## Types
293
-
294
-This document refers to the standard JSON types (e.g., `null`, `string`,
295
-`number`, etc.), as well as a few custom types, described below.
296
-
297
-### `flag`
298
-
299
-Flags allow enabling and disabling features. However, unlike simple booleans,
300
-they can also be `null` (or omitted) to indicate that the default value should
301
-be chosen. This makes it easier for Kubo to change the defaults in the
302
-future unless the user _explicitly_ sets the flag to either `true` (enabled) or
303
-`false` (disabled). Flags have three possible states:
304
-
305
-- `null` or missing (apply the default value).
306
-- `true` (enabled)
307
-- `false` (disabled)
308
-
309
-### `priority`
310
-
311
-Priorities allow specifying the priority of a feature/protocol and disabling the
312
-feature/protocol. Priorities can take one of the following values:
313
-
314
-- `null`/missing (apply the default priority, same as with flags)
315
-- `false` (disabled)
316
-- `1 - 2^63` (priority, lower is preferred)
317
-
318
-### `strings`
319
-
320
-Strings is a special type for conveniently specifying a single string, an array
321
-of strings, or null:
322
-
323
-- `null`
324
-- `"a single string"`
325
-- `["an", "array", "of", "strings"]`
326
-
327
-### `duration`
328
-
329
-Duration is a type for describing lengths of time, using the same format go
330
-does (e.g, `"1d2h4m40.01s"`).
331
-
332
-### `optionalInteger`
333
-
334
-Optional integers allow specifying some numerical value which has
335
-an implicit default when missing from the config file:
336
-
337
-- `null`/missing will apply the default value defined in Kubo sources (`.WithDefault(value)`)
338
-- an integer between `-2^63` and `2^63-1` (i.e. `-9223372036854775808` to `9223372036854775807`)
339
-
340
-### `optionalBytes`
341
-
342
-Optional Bytes allow specifying some number of bytes which has
343
-an implicit default when missing from the config file:
344
-
345
-- `null`/missing (apply the default value defined in Kubo sources)
346
-- a string value indicating the number of bytes, including human readable representations:
347
- - [SI sizes](https://en.wikipedia.org/wiki/Metric_prefix#List_of_SI_prefixes) (metric units, powers of 1000), e.g. `1B`, `2kB`, `3MB`, `4GB`, `5TB`, …)
348
- - [IEC sizes](https://en.wikipedia.org/wiki/Binary_prefix#IEC_prefixes) (binary units, powers of 1024), e.g. `1B`, `2KiB`, `3MiB`, `4GiB`, `5TiB`, …)
349
-
350
-### `optionalString`
351
-
352
-Optional strings allow specifying some string value which has
353
-an implicit default when missing from the config file:
354
-
355
-- `null`/missing will apply the default value defined in Kubo sources (`.WithDefault("value")`)
356
-- a string
357
-
358
-### `optionalDuration`
359
-
360
-Optional durations allow specifying some duration value which has
361
-an implicit default when missing from the config file:
362
-
363
-- `null`/missing will apply the default value defined in Kubo sources (`.WithDefault("1h2m3s")`)
364
-- a string with a valid [go duration](#duration) (e.g, `"1d2h4m40.01s"`).
177
+ - [Profiles](#profiles)
178
+ - [`server` profile](#server-profile)
179
+ - [`randomports` profile](#randomports-profile)
180
+ - [`default-datastore` profile](#default-datastore-profile)
181
+ - [`local-discovery` profile](#local-discovery-profile)
182
+ - [`default-networking` profile](#default-networking-profile)
183
+ - [`flatfs` profile](#flatfs-profile)
184
+ - [`badgerds` profile](#badgerds-profile)
185
+ - [`lowpower` profile](#lowpower-profile)
186
+ - [`legacy-cid-v0` profile](#legacy-cid-v0-profile)
187
+ - [`test-cid-v1` profile](#test-cid-v1-profile)
188
+ - [Types](#types)
189
+ - [`flag`](#flag)
190
+ - [`priority`](#priority)
191
+ - [`strings`](#strings)
192
+ - [`duration`](#duration)
193
+ - [`optionalInteger`](#optionalinteger)
194
+ - [`optionalBytes`](#optionalbytes)
195
+ - [`optionalString`](#optionalstring)
196
+ - [`optionalDuration`](#optionalduration)
197
198
## `Addresses`
199
@@ -2491,3 +2323,194 @@ trigger update warning.
2323
Default: `5`
2324
2325
Type: `optionalInteger` (1-100)
2326
+
2327
+## Profiles
2328
+
2329
+Configuration profiles allow to tweak configuration quickly. Profiles can be
2330
+applied with the `--profile` flag to `ipfs init` or with the `ipfs config profile
2331
+apply` command. When a profile is applied a backup of the configuration file
2332
+will be created in `$IPFS_PATH`.
2333
+
2334
+Configuration profiles can be applied additively. For example, both the `test-cid-v1` and `lowpower` profiles can be applied one after the other.
2335
+The available configuration profiles are listed below. You can also find them
2336
+documented in `ipfs config profile --help`.
2337
+
2338
+### `server` profile
2339
+
2340
+Disables local [`Discovery.MDNS`](#discoverymdns) and blocks connections to
2341
+IPv4 and IPv6 prefixes that are [private, local only, or unrouteable](https://github.com/ipfs/kubo/blob/b71cf0d15904bdef21fe2eee5f1118a274309a4d/config/profile.go#L24-L43).
2342
+
2343
+Recommended when running IPFS on machines with public IPv4 addresses
2344
+at providers that interpret local IPFS discovery and traffic as netscan abuse ([example](https://github.com/ipfs/kubo/issues/10327)).
2345
+
2346
+### `randomports` profile
2347
+
2348
+Use a random port number for the incoming swarm connections.
2349
+Used for testing.
2350
+
2351
+### `default-datastore` profile
2352
+
2353
+Configures the node to use the default datastore (flatfs).
2354
+
2355
+Read the "flatfs" profile description for more information on this datastore.
2356
+
2357
+This profile may only be applied when first initializing the node.
2358
+
2359
+### `local-discovery` profile
2360
+
2361
+Enables local [`Discovery.MDNS`](#discoverymdns) (enabled by default).
2362
+
2363
+Useful to re-enable local discovery after it's disabled by another profile
2364
+(e.g., the server profile).
2365
+
2366
+`test` profile
2367
+
2368
+Reduces external interference of IPFS daemon, this
2369
+is useful when using the daemon in test environments.
2370
+
2371
+### `default-networking` profile
2372
+
2373
+Restores default network settings.
2374
+Inverse profile of the test profile.
2375
+
2376
+### `flatfs` profile
2377
+
2378
+Configures the node to use the flatfs datastore. Flatfs is the default datastore.
2379
+
2380
+This is the most battle-tested and reliable datastore.
2381
+You should use this datastore if:
2382
+
2383
+- You need a very simple and very reliable datastore, and you trust your
2384
+ filesystem. This datastore stores each block as a separate file in the
2385
+ underlying filesystem so it's unlikely to lose data unless there's an issue
2386
+ with the underlying file system.
2387
+- You need to run garbage collection in a way that reclaims free space as soon as possible.
2388
+- You want to minimize memory usage.
2389
+- You are ok with the default speed of data import, or prefer to use `--nocopy`.
2390
+
2391
+This profile may only be applied when first initializing the node.
2392
+
2393
+### `badgerds` profile
2394
+
2395
+Configures the node to use the legacy badgerv1 datastore.
2396
+
2397
+> [!CAUTION]
2398
+> This is based on very old badger 1.x, which has known bugs and is no longer supported by the upstream team.
2399
+> It is provided here only for pre-existing users, allowing them to migrate away to more modern datastore.
2400
+> Do not use it for new deployments, unless you really, really know what you are doing.
2401
+
2402
+Also, be aware that:
2403
+
2404
+- This datastore will not properly reclaim space when your datastore is
2405
+ smaller than several gigabytes. If you run IPFS with `--enable-gc`, you plan on storing very little data in
2406
+ your IPFS node, and disk usage is more critical than performance, consider using
2407
+ `flatfs`.
2408
+- This datastore uses up to several gigabytes of memory.
2409
+- Good for medium-size datastores, but may run into performance issues if your dataset is bigger than a terabyte.
2410
+- The current implementation is based on old badger 1.x which is no longer supported by the upstream team.
2411
+
2412
+This profile may only be applied when first initializing the node.
2413
+
2414
+### `lowpower` profile
2415
+
2416
+Reduces daemon overhead on the system. Affects node
2417
+functionality - performance of content discovery and data
2418
+fetching may be degraded.
2419
+
2420
+> [!CAUTION]
2421
+> Local data won't be announced on routing systems like Amino DHT.
2422
+
2423
+- `Swarm.ConnMgr` set to maintain minimum number of p2p connections at a time.
2424
+- Disables [`Reprovider`](#reprovider) service → no CID will be announced on Amino DHT and other routing systems(!)
2425
+- Disables [`AutoNAT`](#autonat).
2426
+
2427
+Use this profile with caution.
2428
+
2429
+### `legacy-cid-v0` profile
2430
+
2431
+Makes UnixFS import (`ipfs add`) produce legacy CIDv0 with no raw leaves, sha2-256 and 256 KiB chunks.
2432
+
2433
+> [!NOTE]
2434
+> This profile is provided for legacy users and should not be used for new projects.
2435
+
2436
+### `test-cid-v1` profile
2437
+
2438
+Makes UnixFS import (`ipfs add`) produce modern CIDv1 with raw leaves, sha2-256 and 1 MiB chunks.
2439
+
2440
+> [!NOTE]
2441
+> This profile will become the new implicit default, provided for testing purposes.
2442
+> Follow [kubo#4143](https://github.com/ipfs/kubo/issues/4143) for more details.
2443
+
2444
+## Types
2445
+
2446
+This document refers to the standard JSON types (e.g., `null`, `string`,
2447
+`number`, etc.), as well as a few custom types, described below.
2448
+
2449
+### `flag`
2450
+
2451
+Flags allow enabling and disabling features. However, unlike simple booleans,
2452
+they can also be `null` (or omitted) to indicate that the default value should
2453
+be chosen. This makes it easier for Kubo to change the defaults in the
2454
+future unless the user _explicitly_ sets the flag to either `true` (enabled) or
2455
+`false` (disabled). Flags have three possible states:
2456
+
2457
+- `null` or missing (apply the default value).
2458
+- `true` (enabled)
2459
+- `false` (disabled)
2460
+
2461
+### `priority`
2462
+
2463
+Priorities allow specifying the priority of a feature/protocol and disabling the
2464
+feature/protocol. Priorities can take one of the following values:
2465
+
2466
+- `null`/missing (apply the default priority, same as with flags)
2467
+- `false` (disabled)
2468
+- `1 - 2^63` (priority, lower is preferred)
2469
+
2470
+### `strings`
2471
+
2472
+Strings is a special type for conveniently specifying a single string, an array
2473
+of strings, or null:
2474
+
2475
+- `null`
2476
+- `"a single string"`
2477
+- `["an", "array", "of", "strings"]`
2478
+
2479
+### `duration`
2480
+
2481
+Duration is a type for describing lengths of time, using the same format go
2482
+does (e.g, `"1d2h4m40.01s"`).
2483
+
2484
+### `optionalInteger`
2485
+
2486
+Optional integers allow specifying some numerical value which has
2487
+an implicit default when missing from the config file:
2488
+
2489
+- `null`/missing will apply the default value defined in Kubo sources (`.WithDefault(value)`)
2490
+- an integer between `-2^63` and `2^63-1` (i.e. `-9223372036854775808` to `9223372036854775807`)
2491
+
2492
+### `optionalBytes`
2493
+
2494
+Optional Bytes allow specifying some number of bytes which has
2495
+an implicit default when missing from the config file:
2496
+
2497
+- `null`/missing (apply the default value defined in Kubo sources)
2498
+- a string value indicating the number of bytes, including human readable representations:
2499
+ - [SI sizes](https://en.wikipedia.org/wiki/Metric_prefix#List_of_SI_prefixes) (metric units, powers of 1000), e.g. `1B`, `2kB`, `3MB`, `4GB`, `5TB`, …)
2500
+ - [IEC sizes](https://en.wikipedia.org/wiki/Binary_prefix#IEC_prefixes) (binary units, powers of 1024), e.g. `1B`, `2KiB`, `3MiB`, `4GiB`, `5TiB`, …)
2501
+
2502
+### `optionalString`
2503
+
2504
+Optional strings allow specifying some string value which has
2505
+an implicit default when missing from the config file:
2506
+
2507
+- `null`/missing will apply the default value defined in Kubo sources (`.WithDefault("value")`)
2508
+- a string
2509
+
2510
+### `optionalDuration`
2511
+
2512
+Optional durations allow specifying some duration value which has
2513
+an implicit default when missing from the config file:
2514
+
2515
+- `null`/missing will apply the default value defined in Kubo sources (`.WithDefault("1h2m3s")`)
2516
+- a string with a valid [go duration](#duration) (e.g, `"1d2h4m40.01s"`).
docs/datastores.md
+6
@@ -39,6 +39,12 @@ Uses a leveldb database to store key value pairs.
39
40
Uses [badger](https://github.com/dgraph-io/badger) as a key value store.
41
42
+> [!CAUTION]
43
+> This is based on very old badger 1.x, which has known bugs and is no longer supported by the upstream team.
44
+> It is provided here only for pre-existing users, allowing them to migrate away to more modern datastore.
45
+> Do not use it for new deployments, unless you really, really know what you are doing.
46
+
47
+
48
* `syncWrites`: Flush every write to disk before continuing. Setting this to false is safe as kubo will automatically flush writes to disk before and after performing critical operations like pinning. However, you can set this to true to be extra-safe (at the cost of a 2-3x slowdown when adding files).
49
* `truncate`: Truncate the DB if a partially written sector is found (defaults to true). There is no good reason to set this to false unless you want to manually recover partially written (and unpinned) blocks if kubo crashes half-way through adding a file.
50