| 1 | #  IPFS Repo Spec |
| 2 | |
| 3 | **Author(s)**: |
| 4 | - [Juan Benet](github.com/jbenet) |
| 5 | |
| 6 | **Abstract** |
| 7 | |
| 8 | This spec defines an IPFS Repo, its contents, and its interface. It does not specify how the repo data is actually stored, as that is done via swappable implementations. |
| 9 | |
| 10 | # Table of Contents |
| 11 | |
| 12 | - [Definition](#definition) |
| 13 | - [Repo Contents](#repo-contents) |
| 14 | - [version](#version) |
| 15 | - [datastore](#datastore) |
| 16 | - [keystore](#keystore) |
| 17 | - [config (state)](#config-state) |
| 18 | - [locks](#locks) |
| 19 | - [datastore\_spec](#datastore_spec) |
| 20 | - [hooks (TODO)](#hooks-todo) |
| 21 | - [Notes](#notes) |
| 22 | |
| 23 | ## Definition |
| 24 | |
| 25 | A `repo` is the storage repository of an IPFS node. It is the subsystem that |
| 26 | actually stores the data IPFS nodes use. All IPFS objects are stored |
| 27 | in a repo (similar to git). |
| 28 | |
| 29 | There are many possible repo implementations, depending on the storage media |
| 30 | used. Most commonly, IPFS nodes use an [fs-repo](repository_fs.md). |
| 31 | |
| 32 | Repo Implementations: |
| 33 | - [fs-repo](repository_fs.md) - stored in the os filesystem |
| 34 | - mem-repo - stored in process memory |
| 35 | - s3-repo - stored in amazon s3 |
| 36 | |
| 37 | ## Repo Contents |
| 38 | |
| 39 | The Repo stores a collection of [IPLD](https://github.com/ipld/specs#readme) objects that represent: |
| 40 | |
| 41 | - **config** - node configuration and settings |
| 42 | - **datastore** - content stored locally, and indexing data |
| 43 | - **keystore** - cryptographic keys, including node's identity |
| 44 | - **hooks** - scripts to run at predefined times (not yet implemented) |
| 45 | |
| 46 | Note that the IPLD objects a repo stores are divided into: |
| 47 | - **state** (system, control plane) used for the node's internal state |
| 48 | - **content** (userland, data plane) which represent the user's cached and pinned data. |
| 49 | |
| 50 | Additionally, the repo state must determine the following. These need not be IPLD objects, though it is of course encouraged: |
| 51 | |
| 52 | - **version** - the repo version, required for safe migrations |
| 53 | - **locks** - process semaphores for correct concurrent access |
| 54 | - **datastore_spec** - array of mounting points and their properties |
| 55 | |
| 56 | Finally, the repo also stores the blocks with blobs containing binary data. |
| 57 | |
| 58 |  |
| 59 | |
| 60 | ### version |
| 61 | |
| 62 | Repo implementations may change over time, thus they MUST include a `version` recognizable across versions. Meaning that a tool MUST be able to read the `version` of a given repo type. |
| 63 | |
| 64 | For example, the `fs-repo` simply includes a `version` file with the version number. This way, the repo contents can evolve over time but the version remains readable the same way across versions. |
| 65 | |
| 66 | ### datastore |
| 67 | |
| 68 | IPFS nodes store some IPLD objects locally. These are either (a) **state objects** required for local operation -- such as the `config` and `keys` -- or (b) **content objects** used to represent data locally available. **Content objects** are either _pinned_ (stored until they are unpinned) or _cached_ (stored until the next repo garbage collection). |
| 69 | |
| 70 | The name "datastore" comes from [go-datastore](https://github.com/jbenet/go-datastore), a library for swappable key-value stores. Like its name-sake, some repo implementations feature swappable datastores, for example: |
| 71 | - an fs-repo with a leveldb datastore |
| 72 | - an fs-repo with a boltdb datastore |
| 73 | - an fs-repo with a union fs and leveldb datastore |
| 74 | - an fs-repo with an s3 datastore |
| 75 | - an s3-repo with a cached fs and s3 datastore |
| 76 | |
| 77 | This makes it easy to change properties or performance characteristics of a repo without an entirely new implementation. |
| 78 | |
| 79 | ### keystore |
| 80 | |
| 81 | A Repo typically holds the keys a node has access to, for signing and for encryption. |
| 82 | |
| 83 | Details on operation and storage of the keystore can be found in [`repository_fs.md`](repository_fs.md) and [`keystore.md`](keystore.md). |
| 84 | |
| 85 | ### config (state) |
| 86 | |
| 87 | The node's `config` (configuration) is a tree of variables, used to configure various aspects of operation. For example: |
| 88 | - the set of bootstrap peers IPFS uses to connect to the network |
| 89 | - the Swarm, API, and Gateway network listen addresses |
| 90 | - the Datastore configuration regarding the construction and operation of the on-disk storage system. |
| 91 | |
| 92 | There is a set of properties, which are mandatory for the repo usage. Those are `Addresses`, `Discovery`, `Bootstrap`, `Identity`, `Datastore` and `Keychain`. |
| 93 | |
| 94 | It is recommended that `config` files avoid identifying information, so that they may be re-shared across multiple nodes. |
| 95 | |
| 96 | **CHANGES**: today, implementations like js-ipfs and go-ipfs store the peer-id and private key directly in the config. These will be removed and moved out. |
| 97 | |
| 98 | ### locks |
| 99 | |
| 100 | IPFS implementations may use multiple processes, or may disallow multiple processes from using the same repo simultaneously. Others may disallow using the same repo but may allow sharing _datastores_ simultaneously. This synchronization is accomplished via _locks_. |
| 101 | |
| 102 | All repos contain the following standard locks: |
| 103 | - `repo.lock` - prevents concurrent access to the repo. Must be held to _read_ or _write_. |
| 104 | |
| 105 | ### datastore_spec |
| 106 | |
| 107 | This file is created according to the Datastore configuration specified in the `config` file. It contains an array with all the mounting points that the repo is using, as well as its properties. This way, the `datastore_spec` file must have the same mounting points as defined in the Datastore configuration. |
| 108 | |
| 109 | It is important pointing out that the `Datastore` in config must have a `Spec` property, which defines the structure of the ipfs datastore. It is a composable structure, where each datastore is represented by a json object. |
| 110 | |
| 111 | ### hooks (TODO) |
| 112 | |
| 113 | Like git, IPFS nodes will allow `hooks`, a set of user configurable scripts to run at predefined moments in IPFS operations. This makes it easy to customize the behavior of IPFS nodes without changing the implementations themselves. |
| 114 | |
| 115 | ## Notes |
| 116 | |
| 117 | #### A Repo uniquely identifies an IPFS Node |
| 118 | |
| 119 | A repository uniquely identifies a node. Running two different IPFS programs with identical repositories -- and thus identical identities -- WILL cause problems. |
| 120 | |
| 121 | Datastores MAY be shared -- with proper synchronization -- though note that sharing datastore access MAY erode privacy. |
| 122 | |
| 123 | #### Repo implementation changes MUST include migrations |
| 124 | |
| 125 | **DO NOT BREAK USERS' DATA.** This is critical. Thus, any changes to a repo's implementation **MUST** be accompanied by a **SAFE** migration tool. |
| 126 | |
| 127 | See https://github.com/jbenet/go-ipfs/issues/537 and https://github.com/jbenet/random-ideas/issues/33 |
| 128 | |
| 129 | #### Repo Versioning |
| 130 | |
| 131 | A repo version is a single incrementing integer. All versions are considered non-compatible. Repos of different versions MUST be run through the appropriate migration tools before use. |