@cryptotaxi247 / kubo / commits / 48865a909

docs: move kubo-specific docs (#10226)

* docs: move kubo-specific docs * chore: note crypt cmd does not exist Context: https://github.com/ipfs/specs/pull/455 --------- Co-authored-by: Marcin Rataj <lidel@lidel.org>

Henrique Dias committed Nov 20, 2023 at 23:22 UTC 48865a9092d1086952c3f2959b830dfac88ed126
5 files changed +705
docs/specifications/fs-datastore.png
Binary files /dev/null and b/docs/specifications/fs-datastore.png differ
docs/specifications/ipfs-repo-contents.png
Binary files /dev/null and b/docs/specifications/ipfs-repo-contents.png differ
docs/specifications/keystore.md new
+295
@@ -0,0 +1,295 @@
1 +# ![](https://img.shields.io/badge/status-wip-orange.svg?style=flat-square) Keystore
2 +
3 +**Authors(s):**
4 +- [whyrusleeping](github.com/whyrusleeping)
5 +- [Hector Sanjuan](github.com/hsanjuan)
6 +
7 +**Abstract**
8 +
9 +This spec provides definitions and operations for the keystore feature in IPFS.
10 +
11 +# Table of Contents
12 +
13 +- [Goals](#goals)
14 +- [Planned Implementation](#planned-implementation)
15 + - [Key storage](#key-storage)
16 + - [Interface](#interface)
17 + - [Code changes and additions](#code-changes-and-additions)
18 + - [Structures](#structures)
19 +
20 +## Goals
21 +
22 +To have a secure, simple and user-friendly way of storing and managing keys
23 +for use by ipfs. As well as the ability to share these keys, encrypt, decrypt,
24 +sign and verify data.
25 +
26 +## Planned Implementation
27 +
28 +### Key storage
29 +
30 +Storage layout and format is defined in the [`repository_fs`](repository_fs.md) part of the spec.
31 +
32 +### Interface
33 +
34 +#### ipfs key
35 +
36 +```
37 +USAGE
38 + ipfs key - Create and list IPNS name keypairs
39 +
40 + ipfs key
41 +
42 + 'ipfs key gen' generates a new keypair for usage with IPNS and 'ipfs name
43 + publish'.
44 +
45 + > ipfs key gen --type=rsa --size=2048 mykey
46 + > ipfs name publish --key=mykey QmSomeHash
47 +
48 + 'ipfs key list' lists the available keys.
49 +
50 + > ipfs key list
51 + self
52 + mykey
53 +
54 +
55 +SUBCOMMANDS
56 + ipfs key export <name> - Export a keypair
57 + ipfs key gen <name> - Create a new keypair
58 + ipfs key import <name> <key> - Import a key and prints imported key id
59 + ipfs key list - List all local keypairs.
60 + ipfs key rename <name> <newName> - Rename a keypair.
61 + ipfs key rm <name>... - Remove a keypair.
62 + ipfs key rotate - Rotates the IPFS identity.
63 +
64 + For more information about each command, use:
65 + 'ipfs key <subcmd> --help'
66 +```
67 +
68 +#### ipfs crypt
69 +
70 +**NOTE:** as of 2023 Q4, `ipfs crypt` commands are not implemented yet.
71 +
72 +```
73 + ipfs crypt - Perform cryptographic operations using ipfs keypairs
74 +
75 +SUBCOMMANDS:
76 +
77 + ipfs crypt sign <data> - Generates a signature for the given data with a specified key
78 + ipfs crypt verify <data> <sig> - Verify that the given data and signature match
79 + ipfs crypt encrypt <data> - Encrypt the given data
80 + ipfs crypt decrypt <data> - Decrypt the given data
81 +
82 +DESCRIPTION:
83 +
84 + `ipfs crypt` is a command used to perform various cryptographic operations
85 + using ipfs keypairs, including: signing, verifying, encrypting and decrypting.
86 +```
87 +
88 +#### Some subcommands:
89 +
90 +##### ipfs key Gen
91 +
92 +
93 +```
94 +USAGE
95 + ipfs key gen <name> - Create a new keypair
96 +
97 +SYNOPSIS
98 + ipfs key gen [--type=<type> | -t] [--size=<size> | -s]
99 + [--ipns-base=<ipns-base>] [--] <name>
100 +
101 +ARGUMENTS
102 +
103 + <name> - name of key to create
104 +
105 +OPTIONS
106 +
107 + -t, --type string - type of the key to create: rsa, ed25519. Default:
108 + ed25519.
109 + -s, --size int - size of the key to generate.
110 + --ipns-base string - Encoding used for keys: Can either be a multibase
111 + encoded CID or a base58btc encoded multihash. Takes
112 + {b58mh|base36|k|base32|b...}. Default: base36.
113 +```
114 +
115 +* * *
116 +
117 +##### Key Send
118 +
119 +```
120 +USAGE
121 + ipfs key - Create and list IPNS name keypairs
122 +
123 +SYNOPSIS
124 + ipfs key
125 +
126 +DESCRIPTION
127 +
128 + 'ipfs key gen' generates a new keypair for usage with IPNS and 'ipfs name
129 + publish'.
130 +
131 + > ipfs key gen --type=rsa --size=2048 mykey
132 + > ipfs name publish --key=mykey QmSomeHash
133 +
134 + 'ipfs key list' lists the available keys.
135 +
136 + > ipfs key list
137 + self
138 + mykey
139 +
140 +
141 +SUBCOMMANDS
142 + ipfs key export <name> - Export a keypair
143 + ipfs key gen <name> - Create a new keypair
144 + ipfs key import <name> <key> - Import a key and prints imported key id
145 + ipfs key list - List all local keypairs.
146 + ipfs key rename <name> <newName> - Rename a keypair.
147 + ipfs key rm <name>... - Remove a keypair.
148 + ipfs key rotate - Rotates the IPFS identity.
149 +
150 + For more information about each command, use:
151 + 'ipfs key <subcmd> --help'
152 +```
153 +
154 +##### Comments:
155 +
156 +Ensure that the user knows the implications of sending a key.
157 +
158 +* * *
159 +
160 +##### Crypt Encrypt
161 +
162 +```
163 + ipfs crypt encrypt <data> - Encrypt the given data with a specified key
164 +
165 +ARGUMENTS:
166 +
167 + data - The filename of the data to be encrypted ("-" for stdin)
168 +
169 +OPTIONS:
170 +
171 + -k, -key string - The name of the key to use for encryption (default: localkey)
172 + -o, -output string - The name of the output file (default: stdout)
173 + -c, -cipher string - The cipher to use for the operation
174 + -m, -mode string - The block cipher mode to use for the operation
175 +
176 +DESCRIPTION:
177 +
178 + 'ipfs crypt encrypt' is a command used to encypt data so that only holders of a certain
179 + key can read it.
180 +```
181 +
182 +##### Comments:
183 +
184 +This should probably just operate on raw data and not on DAGs.
185 +
186 +* * *
187 +
188 +##### Other Interface Changes
189 +
190 +We will also need to make additions to support keys in other commands, these changes are as follows:
191 +
192 +- `ipfs add`
193 + - Support for a `-encrypt-key` option, for block encrypting the file being added with the key
194 + - also adds an 'encrypted' node above the root unixfs node
195 + - Support for a `-sign-key` option to attach a signature node above the root unixfs node
196 +
197 +- `ipfs block put`
198 + - Support for a `-encrypt-key` option, for encrypting the block before hashing and storing
199 +
200 +- `ipfs object put`
201 + - Support for a `-encrypt-key` option, for encrypting the object before hashing and storing
202 +
203 +- `ipfs name publish`
204 + - Support for a `-key` option to select which keyspace to publish to
205 +
206 +### Code changes and additions
207 +
208 +This sections outlines code organization around this feature.
209 +
210 +#### Keystore package
211 +
212 +The fsrepo carries a `keystore` that can be used to load/store keys. The keystore is implemented following this interface:
213 +
214 +```go
215 +// Keystore provides a key management interface
216 +type Keystore interface {
217 + // Has returns whether or not a key exist in the Keystore
218 + Has(string) (bool, error)
219 + // Put stores a key in the Keystore, if a key with the same name already exists, returns ErrKeyExists
220 + Put(string, ci.PrivKey) error
221 + // Get retrieves a key from the Keystore if it exists, and returns ErrNoSuchKey
222 + // otherwise.
223 + Get(string) (ci.PrivKey, error)
224 + // Delete removes a key from the Keystore
225 + Delete(string) error
226 + // List returns a list of key identifier
227 + List() ([]string, error)
228 +}
229 +```
230 +
231 +Note: Never store passwords as strings, strings cannot be zeroed out after they are used.
232 +using a byte array allows you to write zeroes over the memory so that the users password
233 +does not linger in memory.
234 +
235 +#### Unixfs
236 +
237 +- new node types, 'encrypted' and 'signed', probably shouldn't be in unixfs, just understood by it
238 +- if new node types are not unixfs nodes, special consideration must be given to the interop
239 +
240 +- DagReader needs to be able to access keystore to seamlessly stream encrypted data we have keys for
241 + - also needs to be able to verify signatures
242 +
243 +#### Importer
244 +
245 +- DagBuilderHelper needs to be able to encrypt blocks
246 + - Dag Nodes should be generated like normal, then encrypted, and their parents should
247 + link to the hash of the encrypted node
248 +- DagBuilderParams should have extra parameters to accommodate creating a DBH that encrypts the blocks
249 +
250 +#### New 'Encrypt' package
251 +
252 +Should contain code for crypto operations on dags.
253 +
254 +Encryption of dags should work by first generating a symmetric key, and using
255 +that key to encrypt all the data. That key should then be encrypted with the
256 +public key chosen and stored in the Encrypted DAG structure.
257 +
258 +Note: One option is to simply add it to the key interface.
259 +
260 +### Structures
261 +Some tentative mockups (in json) of the new DAG structures for signing and encrypting
262 +
263 +Signed DAG:
264 +```
265 +{
266 + "Links" : [
267 + {
268 + "Name":"@content",
269 + "Hash":"QmTheContent",
270 + }
271 + ],
272 + "Data": protobuf{
273 + "Type":"Signed DAG",
274 + "Signature": "thesignature",
275 + "PubKeyID": "QmPubKeyHash",
276 + }
277 +}
278 +```
279 +
280 +Encrypted DAG:
281 +```
282 +{
283 + "Links" : [
284 + {
285 + "Name":"@content",
286 + "Hash":"QmRawEncryptedDag",
287 + }
288 + ],
289 + "Data": protobuf{
290 + "Type":"Encrypted DAG",
291 + "PubKeyID": "QmPubKeyHash",
292 + "Key": "ephemeral symmetric key, encrypted with public key",
293 + }
294 +}
295 +```
docs/specifications/repository.md new
+131
@@ -0,0 +1,131 @@
1 +# ![](https://img.shields.io/badge/status-wip-orange.svg?style=flat-square) IPFS Repo Spec
2 +
3 +**Author(s)**:
4 +- [Juan Benet](github.com/jbenet)
5 +
6 +**Abstract**
7 +
8 +This spec defines an IPFS Repo, its contents, and its interface. It does not specify how the repo data is actually stored, as that is done via swappable implementations.
9 +
10 +# Table of Contents
11 +
12 +- [Definition](#definition)
13 +- [Repo Contents](#repo-contents)
14 + - [version](#version)
15 + - [datastore](#datastore)
16 + - [keystore](#keystore)
17 + - [config (state)](#config-state)
18 + - [locks](#locks)
19 + - [datastore\_spec](#datastore_spec)
20 + - [hooks (TODO)](#hooks-todo)
21 +- [Notes](#notes)
22 +
23 +## Definition
24 +
25 +A `repo` is the storage repository of an IPFS node. It is the subsystem that
26 +actually stores the data IPFS nodes use. All IPFS objects are stored
27 +in a repo (similar to git).
28 +
29 +There are many possible repo implementations, depending on the storage media
30 +used. Most commonly, IPFS nodes use an [fs-repo](repository_fs.md).
31 +
32 +Repo Implementations:
33 +- [fs-repo](repository_fs.md) - stored in the os filesystem
34 +- mem-repo - stored in process memory
35 +- s3-repo - stored in amazon s3
36 +
37 +## Repo Contents
38 +
39 +The Repo stores a collection of [IPLD](https://github.com/ipld/specs#readme) objects that represent:
40 +
41 +- **config** - node configuration and settings
42 +- **datastore** - content stored locally, and indexing data
43 +- **keystore** - cryptographic keys, including node's identity
44 +- **hooks** - scripts to run at predefined times (not yet implemented)
45 +
46 +Note that the IPLD objects a repo stores are divided into:
47 +- **state** (system, control plane) used for the node's internal state
48 +- **content** (userland, data plane) which represent the user's cached and pinned data.
49 +
50 +Additionally, the repo state must determine the following. These need not be IPLD objects, though it is of course encouraged:
51 +
52 +- **version** - the repo version, required for safe migrations
53 +- **locks** - process semaphores for correct concurrent access
54 +- **datastore_spec** - array of mounting points and their properties
55 +
56 +Finally, the repo also stores the blocks with blobs containing binary data.
57 +
58 +![](./ipfs-repo-contents.png)
59 +
60 +### version
61 +
62 +Repo implementations may change over time, thus they MUST include a `version` recognizable across versions. Meaning that a tool MUST be able to read the `version` of a given repo type.
63 +
64 +For example, the `fs-repo` simply includes a `version` file with the version number. This way, the repo contents can evolve over time but the version remains readable the same way across versions.
65 +
66 +### datastore
67 +
68 +IPFS nodes store some IPLD objects locally. These are either (a) **state objects** required for local operation -- such as the `config` and `keys` -- or (b) **content objects** used to represent data locally available. **Content objects** are either _pinned_ (stored until they are unpinned) or _cached_ (stored until the next repo garbage collection).
69 +
70 +The name "datastore" comes from [go-datastore](https://github.com/jbenet/go-datastore), a library for swappable key-value stores. Like its name-sake, some repo implementations feature swappable datastores, for example:
71 +- an fs-repo with a leveldb datastore
72 +- an fs-repo with a boltdb datastore
73 +- an fs-repo with a union fs and leveldb datastore
74 +- an fs-repo with an s3 datastore
75 +- an s3-repo with a cached fs and s3 datastore
76 +
77 +This makes it easy to change properties or performance characteristics of a repo without an entirely new implementation.
78 +
79 +### keystore
80 +
81 +A Repo typically holds the keys a node has access to, for signing and for encryption.
82 +
83 +Details on operation and storage of the keystore can be found in [`repository_fs.md`](repository_fs.md) and [`keystore.md`](keystore.md).
84 +
85 +### config (state)
86 +
87 +The node's `config` (configuration) is a tree of variables, used to configure various aspects of operation. For example:
88 +- the set of bootstrap peers IPFS uses to connect to the network
89 +- the Swarm, API, and Gateway network listen addresses
90 +- the Datastore configuration regarding the construction and operation of the on-disk storage system.
91 +
92 +There is a set of properties, which are mandatory for the repo usage. Those are `Addresses`, `Discovery`, `Bootstrap`, `Identity`, `Datastore` and `Keychain`.
93 +
94 +It is recommended that `config` files avoid identifying information, so that they may be re-shared across multiple nodes.
95 +
96 +**CHANGES**: today, implementations like js-ipfs and go-ipfs store the peer-id and private key directly in the config. These will be removed and moved out.
97 +
98 +### locks
99 +
100 +IPFS implementations may use multiple processes, or may disallow multiple processes from using the same repo simultaneously. Others may disallow using the same repo but may allow sharing _datastores_ simultaneously. This synchronization is accomplished via _locks_.
101 +
102 +All repos contain the following standard locks:
103 +- `repo.lock` - prevents concurrent access to the repo. Must be held to _read_ or _write_.
104 +
105 +### datastore_spec
106 +
107 +This file is created according to the Datastore configuration specified in the `config` file. It contains an array with all the mounting points that the repo is using, as well as its properties. This way, the `datastore_spec` file must have the same mounting points as defined in the Datastore configuration.
108 +
109 +It is important pointing out that the `Datastore` in config must have a `Spec` property, which defines the structure of the ipfs datastore. It is a composable structure, where each datastore is represented by a json object.
110 +
111 +### hooks (TODO)
112 +
113 +Like git, IPFS nodes will allow `hooks`, a set of user configurable scripts to run at predefined moments in IPFS operations. This makes it easy to customize the behavior of IPFS nodes without changing the implementations themselves.
114 +
115 +## Notes
116 +
117 +#### A Repo uniquely identifies an IPFS Node
118 +
119 +A repository uniquely identifies a node. Running two different IPFS programs with identical repositories -- and thus identical identities -- WILL cause problems.
120 +
121 +Datastores MAY be shared -- with proper synchronization -- though note that sharing datastore access MAY erode privacy.
122 +
123 +#### Repo implementation changes MUST include migrations
124 +
125 +**DO NOT BREAK USERS' DATA.** This is critical. Thus, any changes to a repo's implementation **MUST** be accompanied by a **SAFE** migration tool.
126 +
127 +See https://github.com/jbenet/go-ipfs/issues/537 and https://github.com/jbenet/random-ideas/issues/33
128 +
129 +#### Repo Versioning
130 +
131 +A repo version is a single incrementing integer. All versions are considered non-compatible. Repos of different versions MUST be run through the appropriate migration tools before use.
docs/specifications/repository_fs.md new
+279
@@ -0,0 +1,279 @@
1 +# ![](https://img.shields.io/badge/status-wip-orange.svg?style=flat-square) fs-repo
2 +
3 +**Author(s)**:
4 +- [Juan Benet](github.com/jbenet)
5 +- [David Dias](github.com/daviddias)
6 +- [Hector Sanjuan](github.com/hsanjuan)
7 +
8 +**Abstract**
9 +
10 +This spec defines `fs-repo` version `1`, its formats, and semantics.
11 +
12 +# Table of Contents
13 +
14 +- [Definition](#definition)
15 +- [Contents](#contents)
16 + - [api](#api)
17 + - [blocks/](#blocks)
18 + - [config](#config)
19 + - [hooks/](#hooks)
20 + - [keystore/](#keystore)
21 + - [datastore/](#datastore)
22 + - [logs/](#logs)
23 + - [repo.lock](#repolock)
24 + - [version](#version)
25 +- [Datastore](#datastore-1)
26 +- [Notes](#notes)
27 + - [Location](#location)
28 + - [blocks/ with an fs-datastore](#blocks-with-an-fs-datastore)
29 + - [Reading without the `repo.lock`](#reading-without-the-repolock)
30 +
31 +## Definition
32 +
33 +`fs-repo` is a filesystem implementation of the IPFS [repo](repository.md).
34 +
35 +
36 +## Contents
37 +
38 +![](img/ipfs-repo-contents.png?)
39 +
40 +```
41 +.ipfs/
42 +├── api <--- running daemon api addr
43 +├── blocks/ <--- objects stored directly on disk
44 +│ └── aa <--- prefix namespacing like git
45 +│ └── aa <--- N tiers
46 +├── config <--- config file (json or toml)
47 +├── hooks/ <--- hook scripts
48 +├── keystore/ <--- cryptographic keys
49 +│ ├── key_b32name <--- private key with base32-encoded name
50 +├── datastore/ <--- datastore
51 +├── logs/ <--- 1 or more files (log rotate)
52 +│ └── events.log <--- can be tailed
53 +├── repo.lock <--- mutex for repo
54 +└── version <--- version file
55 +```
56 +
57 +### api
58 +
59 +`./api` is a file that exists to denote an API endpoint to listen to.
60 +- It MAY exist even if the endpoint is no longer live (i.e. it is a _stale_ or left-over `./api` file).
61 +
62 +In the presence of an `./api` file, ipfs tools (e.g. go-ipfs `ipfs daemon`) MUST attempt to delegate to the endpoint, and MAY remove the file if reasonably certain the file is stale. (e.g. endpoint is local, but no process is live)
63 +
64 +The `./api` file is used in conjunction with the `repo.lock`. Clients may opt to use the api service, or wait until the process holding `repo.lock` exits. The file's content is the api endpoint as a [multiaddr](https://github.com/jbenet/multiaddr)
65 +
66 +```
67 +> cat .ipfs/api
68 +/ip4/127.0.0.1/tcp/5001
69 +```
70 +
71 +Notes:
72 +- The API server must remove the api file before releasing the `repo.lock`.
73 +- It is not enough to use the `config` file, as the API addr of a daemon may
74 + have been overridden via ENV or flag.
75 +
76 +#### api file for remote control
77 +
78 +One use case of the `api` file is to have a repo directory like:
79 +
80 +```
81 +> tree $IPFS_PATH
82 +/Users/jbenet/.ipfs
83 +└── api
84 +
85 +0 directories, 1 files
86 +
87 +> cat $IPFS_PATH/api
88 +/ip4/1.2.3.4/tcp/5001
89 +```
90 +
91 +In go-ipfs, this has the same effect as:
92 +
93 +```
94 +ipfs --api /ip4/1.2.3.4/tcp/5001 <cmd>
95 +```
96 +
97 +Meaning that it makes ipfs tools use an ipfs node at the given endpoint, instead of the local directory as a repo.
98 +
99 +In this use case, the rest of the `$IPFS_PATH` may be completely empty, and no other information is necessary. It cannot be said it is a _repo_ per-se. (TODO: come up with a good name for this).
100 +
101 +### blocks/
102 +
103 +The `block/` component contains the raw data representing all IPFS objects
104 +stored locally, whether pinned or cached. This component is controlled by the `
105 +datastore`. For example, it may be stored within a leveldb instance in `
106 +datastore/`, or it may be stored entirely with independent files, like git.
107 +
108 +In the default case, the user uses fs-datastore for all `/blocks` so the
109 +objects are stored in individual files. In other cases, `/blocks` may even be
110 +stored remotely
111 +
112 +- [blocks/ with an fs-datastore](#blocks-with-an-fs-datastore)
113 +
114 +### config
115 +
116 +The `config` file is a JSON or TOML file that contains the tree of
117 +configuration variables. It MUST only be changed while holding the
118 +`repo.lock`, or potentially lose edits.
119 +
120 +### hooks/
121 +
122 +The `hooks` directory contains executable scripts to be called on specific
123 +events to alter ipfs node behavior.
124 +
125 +Currently available hooks:
126 +
127 +```
128 +none
129 +```
130 +
131 +### keystore/
132 +
133 +
134 +The `keystore` directory holds additional private keys that the node has
135 +access to (the public keys can be derived from them).
136 +
137 +The keystore repository should have `0700` permissions (readable, writable by
138 +the owner only).
139 +
140 +The key files are named as `key_base32encodedNameNoPadding` where `key_` is a
141 +fixed prefix followed by a base32 encoded identifier, **without padding and
142 +downcased**. The identifier usually corresponds to a human-friendly name given
143 +by the user.
144 +
145 +The key files should have '0400' permissions (read-only, by the owner only).
146 +
147 +The `self` key identifier is reserved for the peer's main key, and therefore key named
148 +`key_onswyzq` is allowed in this folder.
149 +
150 +The key files themselves contain a serialized representation of the keys as
151 +defined in the
152 +[libp2p specification](https://github.com/libp2p/specs/blob/master/peer-ids/peer-ids.md#keys).
153 +
154 +### datastore/
155 +
156 +The `datastore` directory contains the data for a leveldb instance used to
157 +store operation data for the IPFS node. If the user uses a `boltdb` datastore
158 +instead, the directory will be named `boltdb`. Thus the data files of each
159 +database will not clash.
160 +
161 +TODO: consider whether all should just be named `leveldb/`
162 +
163 +### logs/
164 +
165 +IPFS implementations put event log files inside the `logs/` directory. The
166 +latest log file is `logs/events`. Others, rotated out may exist, with a
167 +timestamp of their creation. For example:
168 +
169 +
170 +
171 +### repo.lock
172 +
173 +`repo.lock` prevents concurrent access to the repo. Its content SHOULD BE the
174 +PID of the process currently holding the lock. This allows clients to detect
175 +a failed lock and cleanup.
176 +
177 +```
178 +> cat .ipfs/repo.lock
179 +42
180 +> ps | grep "ipfs daemon"
181 +42 ttys000 79:05.83 ipfs daemon
182 +```
183 +
184 +**TODO, ADDRESS DISCREPANCY:** the go-ipfs implementation does not currently store the PID in the file, which in some systems causes failures after a failure or a teardown. This SHOULD NOT require any manual intervention-- a present lock should give new processes enough information to recover. Doing this correctly in a portable, safe way, with good UX is very tricky. We must be careful with TOCTTOU bugs, and multiple concurrent processes capable of running at any moment. The goal is for all processes to operate safely, to avoid bothering the user, and for the repo to always remain in a correct, consistent state.
185 +
186 +### version
187 +
188 +The `version` file contains the repo implementation name and version. This format has changed over time:
189 +
190 +```
191 +# in version 0
192 +> cat $repo-at-version-0/version
193 +cat: /Users/jbenet/.ipfs/version: No such file or directory
194 +
195 +# in versions 1 and 2
196 +> cat $repo-at-version-1/version
197 +1
198 +> cat $repo-at-version-2/version
199 +2
200 +
201 +# in versions >3
202 +> cat $repo-at-version-3/version
203 +fs-repo/3
204 +```
205 +
206 +_Any_ fs-repo implementation of _any_ versions `>0` MUST be able to read the
207 +`version` file. It MUST NOT change format between versions. The sole exception is version 0, which had no file.
208 +
209 +**TODO: ADDRESS DISCREPANCY:** versions 1 and 2 of the go-ipfs implementation use just the integer number. It SHOULD have used `fs-repo/<version-number>`. We could either change the spec and always just use the int, or change go-ipfs in version `>3`. we will have to be backwards compatible.
210 +
211 +## Datastore
212 +
213 +Both the `/blocks` and `/datastore` directories are controlled by the
214 +`datastore` component of the repo.
215 +
216 +## Notes
217 +
218 +### Location
219 +
220 +The `fs-repo` can be located anywhere on the filesystem. By default
221 +clients should search for a repo in:
222 +
223 +```
224 +~/.ipfs
225 +```
226 +
227 +Users can tell IPFS programs to look elsewhere with the env var:
228 +
229 +```
230 +IPFS_PATH=/path/to/repo
231 +```
232 +
233 +### blocks/ with an fs-datastore
234 +
235 +![](fs-datastore.png)
236 +
237 +Each object is stored in its own file. The filename is the hash of the object.
238 +The files are nested in directories whose names are prefixes of the hash, as
239 +in `.git/objects`.
240 +
241 +For example:
242 +```sh
243 +# multihashes
244 +1220fe389b55ea958590769f9046b0f7268bca90a92e4a9f45cbb30930f4bf89269d # sha2
245 +1114f623e0ec7f8719fb14a18838d2a3ef4e550b5e53 # sha1
246 +
247 +# locations of the blocks
248 +.ipfs/blocks/1114/f6/23/e0ec7f8719fb14a18838d2a3ef4e550b5e53
249 +.ipfs/blocks/1220/fe/38/9b55ea958590769f9046b0f7268bca90a92e4a9f45cbb30930f4bf89269d
250 +```
251 +
252 +**Important Notes:**
253 +- the hashes are encoded in hex, not the usual base58, because some
254 + filesystems are case insensitive.
255 +- the multihash prefix is two bytes, which would waste two directory levels,
256 + thus these are combined into one.
257 +- the git `idx` and `pack` file formats could be used to coalesce objects
258 +
259 +**TODO: ADDRESS DISCREPANCY:**
260 +
261 +the go-ipfs fs-repo in version 2 uses a different `blocks/` dir layout:
262 +
263 +```
264 +/Users/jbenet/.ipfs/blocks
265 +├── 12200007
266 +│ └── 12200007d4e3a319cd8c7c9979280e150fc5dbaae1ce54e790f84ae5fd3c3c1a0475.data
267 +├── 1220000f
268 +│ └── 1220000fadd95a98f3a47c1ba54a26c77e15c1a175a975d88cf198cc505a06295b12.data
269 +```
270 +
271 +We MUST address whether we should change the fs-repo spec to match go-ipfs in version 2, or we should change go-ipfs to match the fs-repo spec (more tiers). We MUST also address whether the levels are a repo version parameter or a config parameter. There are filesystems in which a different fanout will have wildly different performance. These are mostly networked and legacy filesystems.
272 +
273 +### Reading without the `repo.lock`
274 +
275 +Programs MUST hold the `repo.lock` while reading and writing most files in the
276 +repo. The only two exceptions are:
277 +
278 +- `repo.lock` - so clients may check for it
279 +- `api` - so clients may use the API