| 1 | #  fs-repo |
| 2 | |
| 3 | **Author(s)**: |
| 4 | - [Juan Benet](github.com/jbenet) |
| 5 | - [David Dias](github.com/daviddias) |
| 6 | - [Hector Sanjuan](github.com/hsanjuan) |
| 7 | |
| 8 | **Abstract** |
| 9 | |
| 10 | This spec defines `fs-repo` version `1`, its formats, and semantics. |
| 11 | |
| 12 | # Table of Contents |
| 13 | |
| 14 | - [Definition](#definition) |
| 15 | - [Contents](#contents) |
| 16 | - [api](#api) |
| 17 | - [blocks/](#blocks) |
| 18 | - [config](#config) |
| 19 | - [hooks/](#hooks) |
| 20 | - [keystore/](#keystore) |
| 21 | - [datastore/](#datastore) |
| 22 | - [logs/](#logs) |
| 23 | - [repo.lock](#repolock) |
| 24 | - [version](#version) |
| 25 | - [Datastore](#datastore-1) |
| 26 | - [Notes](#notes) |
| 27 | - [Location](#location) |
| 28 | - [blocks/ with an fs-datastore](#blocks-with-an-fs-datastore) |
| 29 | - [Reading without the `repo.lock`](#reading-without-the-repolock) |
| 30 | |
| 31 | ## Definition |
| 32 | |
| 33 | `fs-repo` is a filesystem implementation of the IPFS [repo](repository.md). |
| 34 | |
| 35 | |
| 36 | ## Contents |
| 37 | |
| 38 |  |
| 39 | |
| 40 | ``` |
| 41 | .ipfs/ |
| 42 | ├── api <--- running daemon api addr |
| 43 | ├── blocks/ <--- objects stored directly on disk |
| 44 | │ └── aa <--- prefix namespacing like git |
| 45 | │ └── aa <--- N tiers |
| 46 | ├── config <--- config file (json or toml) |
| 47 | ├── hooks/ <--- hook scripts |
| 48 | ├── keystore/ <--- cryptographic keys |
| 49 | │ ├── key_b32name <--- private key with base32-encoded name |
| 50 | ├── datastore/ <--- datastore |
| 51 | ├── logs/ <--- 1 or more files (log rotate) |
| 52 | │ └── events.log <--- can be tailed |
| 53 | ├── repo.lock <--- mutex for repo |
| 54 | └── version <--- version file |
| 55 | ``` |
| 56 | |
| 57 | ### api |
| 58 | |
| 59 | `./api` is a file that exists to denote an API endpoint to listen to. |
| 60 | - It MAY exist even if the endpoint is no longer live (i.e. it is a _stale_ or left-over `./api` file). |
| 61 | |
| 62 | In the presence of an `./api` file, ipfs tools (e.g. go-ipfs `ipfs daemon`) MUST attempt to delegate to the endpoint, and MAY remove the file if reasonably certain the file is stale. (e.g. endpoint is local, but no process is live) |
| 63 | |
| 64 | The `./api` file is used in conjunction with the `repo.lock`. Clients may opt to use the api service, or wait until the process holding `repo.lock` exits. The file's content is the api endpoint as a [multiaddr](https://github.com/jbenet/multiaddr) |
| 65 | |
| 66 | ``` |
| 67 | > cat .ipfs/api |
| 68 | /ip4/127.0.0.1/tcp/5001 |
| 69 | ``` |
| 70 | |
| 71 | Notes: |
| 72 | - The API server must remove the api file before releasing the `repo.lock`. |
| 73 | - It is not enough to use the `config` file, as the API addr of a daemon may |
| 74 | have been overridden via ENV or flag. |
| 75 | |
| 76 | #### api file for remote control |
| 77 | |
| 78 | One use case of the `api` file is to have a repo directory like: |
| 79 | |
| 80 | ``` |
| 81 | > tree $IPFS_PATH |
| 82 | /Users/jbenet/.ipfs |
| 83 | └── api |
| 84 | |
| 85 | 0 directories, 1 files |
| 86 | |
| 87 | > cat $IPFS_PATH/api |
| 88 | /ip4/1.2.3.4/tcp/5001 |
| 89 | ``` |
| 90 | |
| 91 | In go-ipfs, this has the same effect as: |
| 92 | |
| 93 | ``` |
| 94 | ipfs --api /ip4/1.2.3.4/tcp/5001 <cmd> |
| 95 | ``` |
| 96 | |
| 97 | Meaning that it makes ipfs tools use an ipfs node at the given endpoint, instead of the local directory as a repo. |
| 98 | |
| 99 | In this use case, the rest of the `$IPFS_PATH` may be completely empty, and no other information is necessary. It cannot be said it is a _repo_ per-se. (TODO: come up with a good name for this). |
| 100 | |
| 101 | ### blocks/ |
| 102 | |
| 103 | The `block/` component contains the raw data representing all IPFS objects |
| 104 | stored locally, whether pinned or cached. This component is controlled by the ` |
| 105 | datastore`. For example, it may be stored within a leveldb instance in ` |
| 106 | datastore/`, or it may be stored entirely with independent files, like git. |
| 107 | |
| 108 | In the default case, the user uses fs-datastore for all `/blocks` so the |
| 109 | objects are stored in individual files. In other cases, `/blocks` may even be |
| 110 | stored remotely |
| 111 | |
| 112 | - [blocks/ with an fs-datastore](#blocks-with-an-fs-datastore) |
| 113 | |
| 114 | ### config |
| 115 | |
| 116 | The `config` file is a JSON or TOML file that contains the tree of |
| 117 | configuration variables. It MUST only be changed while holding the |
| 118 | `repo.lock`, or potentially lose edits. |
| 119 | |
| 120 | ### hooks/ |
| 121 | |
| 122 | The `hooks` directory contains executable scripts to be called on specific |
| 123 | events to alter ipfs node behavior. |
| 124 | |
| 125 | Currently available hooks: |
| 126 | |
| 127 | ``` |
| 128 | none |
| 129 | ``` |
| 130 | |
| 131 | ### keystore/ |
| 132 | |
| 133 | |
| 134 | The `keystore` directory holds additional private keys that the node has |
| 135 | access to (the public keys can be derived from them). |
| 136 | |
| 137 | The keystore repository should have `0700` permissions (readable, writable by |
| 138 | the owner only). |
| 139 | |
| 140 | The key files are named as `key_base32encodedNameNoPadding` where `key_` is a |
| 141 | fixed prefix followed by a base32 encoded identifier, **without padding and |
| 142 | downcased**. The identifier usually corresponds to a human-friendly name given |
| 143 | by the user. |
| 144 | |
| 145 | The key files should have '0400' permissions (read-only, by the owner only). |
| 146 | |
| 147 | The `self` key identifier is reserved for the peer's main key, and therefore key named |
| 148 | `key_onswyzq` is allowed in this folder. |
| 149 | |
| 150 | The key files themselves contain a serialized representation of the keys as |
| 151 | defined in the |
| 152 | [libp2p specification](https://github.com/libp2p/specs/blob/master/peer-ids/peer-ids.md#keys). |
| 153 | |
| 154 | ### datastore/ |
| 155 | |
| 156 | The `datastore` directory contains the data for a leveldb instance used to |
| 157 | store operation data for the IPFS node. If the user uses a `boltdb` datastore |
| 158 | instead, the directory will be named `boltdb`. Thus the data files of each |
| 159 | database will not clash. |
| 160 | |
| 161 | TODO: consider whether all should just be named `leveldb/` |
| 162 | |
| 163 | ### logs/ |
| 164 | |
| 165 | IPFS implementations put event log files inside the `logs/` directory. The |
| 166 | latest log file is `logs/events`. Others, rotated out may exist, with a |
| 167 | timestamp of their creation. For example: |
| 168 | |
| 169 | |
| 170 | |
| 171 | ### repo.lock |
| 172 | |
| 173 | `repo.lock` prevents concurrent access to the repo. Its content SHOULD BE the |
| 174 | PID of the process currently holding the lock. This allows clients to detect |
| 175 | a failed lock and cleanup. |
| 176 | |
| 177 | ``` |
| 178 | > cat .ipfs/repo.lock |
| 179 | 42 |
| 180 | > ps | grep "ipfs daemon" |
| 181 | 42 ttys000 79:05.83 ipfs daemon |
| 182 | ``` |
| 183 | |
| 184 | **TODO, ADDRESS DISCREPANCY:** the go-ipfs implementation does not currently store the PID in the file, which in some systems causes failures after a failure or a teardown. This SHOULD NOT require any manual intervention-- a present lock should give new processes enough information to recover. Doing this correctly in a portable, safe way, with good UX is very tricky. We must be careful with TOCTTOU bugs, and multiple concurrent processes capable of running at any moment. The goal is for all processes to operate safely, to avoid bothering the user, and for the repo to always remain in a correct, consistent state. |
| 185 | |
| 186 | ### version |
| 187 | |
| 188 | The `version` file contains the repo implementation name and version. This format has changed over time: |
| 189 | |
| 190 | ``` |
| 191 | # in version 0 |
| 192 | > cat $repo-at-version-0/version |
| 193 | cat: /Users/jbenet/.ipfs/version: No such file or directory |
| 194 | |
| 195 | # in versions 1 and 2 |
| 196 | > cat $repo-at-version-1/version |
| 197 | 1 |
| 198 | > cat $repo-at-version-2/version |
| 199 | 2 |
| 200 | |
| 201 | # in versions >3 |
| 202 | > cat $repo-at-version-3/version |
| 203 | fs-repo/3 |
| 204 | ``` |
| 205 | |
| 206 | _Any_ fs-repo implementation of _any_ versions `>0` MUST be able to read the |
| 207 | `version` file. It MUST NOT change format between versions. The sole exception is version 0, which had no file. |
| 208 | |
| 209 | **TODO: ADDRESS DISCREPANCY:** versions 1 and 2 of the go-ipfs implementation use just the integer number. It SHOULD have used `fs-repo/<version-number>`. We could either change the spec and always just use the int, or change go-ipfs in version `>3`. we will have to be backwards compatible. |
| 210 | |
| 211 | ## Datastore |
| 212 | |
| 213 | Both the `/blocks` and `/datastore` directories are controlled by the |
| 214 | `datastore` component of the repo. |
| 215 | |
| 216 | ## Notes |
| 217 | |
| 218 | ### Location |
| 219 | |
| 220 | The `fs-repo` can be located anywhere on the filesystem. By default |
| 221 | clients should search for a repo in: |
| 222 | |
| 223 | ``` |
| 224 | ~/.ipfs |
| 225 | ``` |
| 226 | |
| 227 | Users can tell IPFS programs to look elsewhere with the env var: |
| 228 | |
| 229 | ``` |
| 230 | IPFS_PATH=/path/to/repo |
| 231 | ``` |
| 232 | |
| 233 | ### blocks/ with an fs-datastore |
| 234 | |
| 235 |  |
| 236 | |
| 237 | Each object is stored in its own file. The filename is the hash of the object. |
| 238 | The files are nested in directories whose names are prefixes of the hash, as |
| 239 | in `.git/objects`. |
| 240 | |
| 241 | For example: |
| 242 | ```sh |
| 243 | # multihashes |
| 244 | 1220fe389b55ea958590769f9046b0f7268bca90a92e4a9f45cbb30930f4bf89269d # sha2 |
| 245 | 1114f623e0ec7f8719fb14a18838d2a3ef4e550b5e53 # sha1 |
| 246 | |
| 247 | # locations of the blocks |
| 248 | .ipfs/blocks/1114/f6/23/e0ec7f8719fb14a18838d2a3ef4e550b5e53 |
| 249 | .ipfs/blocks/1220/fe/38/9b55ea958590769f9046b0f7268bca90a92e4a9f45cbb30930f4bf89269d |
| 250 | ``` |
| 251 | |
| 252 | **Important Notes:** |
| 253 | - the hashes are encoded in hex, not the usual base58, because some |
| 254 | filesystems are case insensitive. |
| 255 | - the multihash prefix is two bytes, which would waste two directory levels, |
| 256 | thus these are combined into one. |
| 257 | - the git `idx` and `pack` file formats could be used to coalesce objects |
| 258 | |
| 259 | **TODO: ADDRESS DISCREPANCY:** |
| 260 | |
| 261 | the go-ipfs fs-repo in version 2 uses a different `blocks/` dir layout: |
| 262 | |
| 263 | ``` |
| 264 | /Users/jbenet/.ipfs/blocks |
| 265 | ├── 12200007 |
| 266 | │ └── 12200007d4e3a319cd8c7c9979280e150fc5dbaae1ce54e790f84ae5fd3c3c1a0475.data |
| 267 | ├── 1220000f |
| 268 | │ └── 1220000fadd95a98f3a47c1ba54a26c77e15c1a175a975d88cf198cc505a06295b12.data |
| 269 | ``` |
| 270 | |
| 271 | We MUST address whether we should change the fs-repo spec to match go-ipfs in version 2, or we should change go-ipfs to match the fs-repo spec (more tiers). We MUST also address whether the levels are a repo version parameter or a config parameter. There are filesystems in which a different fanout will have wildly different performance. These are mostly networked and legacy filesystems. |
| 272 | |
| 273 | ### Reading without the `repo.lock` |
| 274 | |
| 275 | Programs MUST hold the `repo.lock` while reading and writing most files in the |
| 276 | repo. The only two exceptions are: |
| 277 | |
| 278 | - `repo.lock` - so clients may check for it |
| 279 | - `api` - so clients may use the API |