| 1 | # IPFS API Implementation Doc |
| 2 | |
| 3 | This short document aims to give a quick guide to anyone implementing API |
| 4 | bindings for IPFS implementations-- in particular kubo. |
| 5 | |
| 6 | Sections: |
| 7 | - IPFS Types |
| 8 | - API Transports |
| 9 | - API Commands |
| 10 | - Implementing bindings for the HTTP API |
| 11 | |
| 12 | ## IPFS Types |
| 13 | |
| 14 | IPFS uses a set of value type that is useful to enumerate up front: |
| 15 | |
| 16 | - `<ipfs-path>` is unix-style path, beginning with `/ipfs/<cid>/...` or |
| 17 | `/ipns/<hash>/...` or `/ipns/<domain>/...`. |
| 18 | - `<hash>` is a base58 encoded [multihash](https://github.com/multiformats/multihash) |
| 19 | - `cid` is a [multibase](https://github.com/multiformats/multibase) encoded |
| 20 | [CID](https://github.com/ipld/cid) - a self-describing content-addressing identifier |
| 21 | |
| 22 | A note on streams: IPFS is a streaming protocol. Everything about it can be |
| 23 | streamed. When importing files, API requests should aim to stream the data in, |
| 24 | and handle back-pressure correctly, so that the IPFS node can handle it |
| 25 | sequentially without too much memory pressure. (If using HTTP, this is typically |
| 26 | handled for you by writes to the request body blocking.) |
| 27 | |
| 28 | ## API Transports |
| 29 | |
| 30 | Like with everything else, IPFS aims to be flexible regarding the API transports. |
| 31 | Currently, the [kubo](https://github.com/ipfs/kubo) implementation supports |
| 32 | both an in-process API and an HTTP API. More can be added easily, by mapping the |
| 33 | API functions over a transport. (This is similar to how gRPC is also _mapped on |
| 34 | top of transports_, like HTTP). |
| 35 | |
| 36 | Mapping to a transport involves leveraging the transport's features to express |
| 37 | function calls. For example: |
| 38 | |
| 39 | #### CLI API Transport |
| 40 | |
| 41 | In the commandline, IPFS uses a traditional flag and arg-based mapping, where: |
| 42 | - the first arguments select the command, as in git - e.g. `ipfs dag get` |
| 43 | - the flags specify options - e.g. `--enc=protobuf -q` |
| 44 | - the rest are positional arguments - e.g. `ipfs key rename <name> <newName>` |
| 45 | - files are specified by filename, or through stdin |
| 46 | |
| 47 | (NOTE: When kubo runs the daemon, the CLI API is converted to HTTP |
| 48 | calls. otherwise, they execute in the same process) |
| 49 | |
| 50 | #### HTTP API Transport |
| 51 | |
| 52 | In HTTP, our API layering uses a REST-like mapping, where: |
| 53 | - the URL path selects the command - e.g `/object/get` |
| 54 | - the URL query string implements option arguments - e.g. `&enc=protobuf&q=true` |
| 55 | - the URL query also implements positional arguments - e.g. |
| 56 | `&arg=<hash1>&arg=add-link&arg=foo&arg=<hash2>` |
| 57 | - the request body streams file data - reads files or stdin |
| 58 | - multiple streams are muxed with multipart (todo: add tar stream support) |
| 59 | |
| 60 | |
| 61 | ## API Commands |
| 62 | |
| 63 | There is a "standard IPFS API" which is currently defined as "all the commands |
| 64 | exposed by the kubo implementation". There are auto-generated [API Docs](https://ipfs.io/docs/api/). |
| 65 | You can Also see [a listing here](https://github.com/ipfs/kubo/blob/94b832df861728c65e912935641d08880c341e0a/core/commands/root.go#L96-L130), or get a list of |
| 66 | commands by running `ipfs commands` locally. |
| 67 | |
| 68 | ## Implementing bindings for the HTTP API |
| 69 | |
| 70 | As mentioned above, the API commands map to HTTP with: |
| 71 | - the URL path selects the command - e.g `/object/get` |
| 72 | - the URL query string implements option arguments - e.g. `&enc=protobuf&q=true` |
| 73 | - the URL query also implements positional arguments - e.g. |
| 74 | `&arg=<hash1>&arg=add-link&arg=foo&arg=<hash2>` |
| 75 | - the request body streams file data - reads files or stdin |
| 76 | - multiple streams are muxed with multipart (todo: add tar stream support) |
| 77 | |
| 78 | You can see the latest [list of our HTTP RPC clients here](http-rpc-clients.md) |
| 79 | |
| 80 | The Go implementation is good to answer harder questions, like how is multipart |
| 81 | handled, or what headers should be set in edge conditions. But the javascript |
| 82 | implementation is very concise, and easy to follow. |
| 83 | |
| 84 | ## Note on multipart + inspecting requests |
| 85 | |
| 86 | Despite all the generalization spoken about above, the IPFS API is actually very |
| 87 | simple. You can inspect all the requests made with `nc` and the `--api` option |
| 88 | (as of [this PR](https://github.com/ipfs/kubo/pull/1598), or `0.3.8`): |
| 89 | |
| 90 | ```sh |
| 91 | > nc -l 5002 & |
| 92 | > ipfs --api /ip4/127.0.0.1/tcp/5002 swarm addrs local --enc=json |
| 93 | POST /api/v0/version?enc=json&stream-channels=true HTTP/1.1 |
| 94 | Host: 127.0.0.1:5002 |
| 95 | User-Agent: /kubo/0.14.0/ |
| 96 | Content-Length: 0 |
| 97 | Content-Type: application/octet-stream |
| 98 | Accept-Encoding: gzip |
| 99 | |
| 100 | |
| 101 | ``` |
| 102 | |
| 103 | The only hard part is getting the file streaming right. It is (now) fairly easy |
| 104 | to stream files to kubo using multipart. Basically, we end up with HTTP |
| 105 | requests like this: |
| 106 | |
| 107 | ```sh |
| 108 | > nc -l 5002 & |
| 109 | > ipfs --api /ip4/127.0.0.1/tcp/5002 add -r ~/demo/basic/test |
| 110 | POST /api/v0/add?encoding=json&progress=true&r=true&stream-channels=true HTTP/1.1 |
| 111 | Host: 127.0.0.1:5002 |
| 112 | User-Agent: /kubo/0.14.0/ |
| 113 | Transfer-Encoding: chunked |
| 114 | Content-Disposition: form-data: name="files" |
| 115 | Content-Type: multipart/form-data; boundary=2186ef15d8f2c4f100af72d6d345afe36a4d17ef11264ec5b8ec4436447f |
| 116 | Accept-Encoding: gzip |
| 117 | |
| 118 | 1 |
| 119 | - |
| 120 | e5 |
| 121 | -2186ef15d8f2c4f100af72d6d345afe36a4d17ef11264ec5b8ec4436447f |
| 122 | Content-Disposition: form-data; name="file"; filename="test" |
| 123 | Content-Type: multipart/mixed; boundary=acdb172fe12f25e8ffae9981ce6f4580abdefb0cae3ceebe464d802866be |
| 124 | |
| 125 | |
| 126 | 9c |
| 127 | --acdb172fe12f25e8ffae9981ce6f4580abdefb0cae3ceebe464d802866be |
| 128 | Content-Disposition: file; filename="test%2Fbar" |
| 129 | Content-Type: application/octet-stream |
| 130 | |
| 131 | |
| 132 | 4 |
| 133 | bar |
| 134 | |
| 135 | dc |
| 136 | |
| 137 | --acdb172fe12f25e8ffae9981ce6f4580abdefb0cae3ceebe464d802866be |
| 138 | Content-Disposition: file; filename="test%2Fbaz" |
| 139 | Content-Type: multipart/mixed; boundary=2799ac77a72ef7b8a0281945806b9f9a28f7681145aa8e91b052d599b2dd |
| 140 | |
| 141 | |
| 142 | a0 |
| 143 | --2799ac77a72ef7b8a0281945806b9f9a28f7681145aa8e91b052d599b2dd |
| 144 | Content-Type: application/octet-stream |
| 145 | Content-Disposition: file; filename="test%2Fbaz%2Fb" |
| 146 | |
| 147 | |
| 148 | 4 |
| 149 | bar |
| 150 | |
| 151 | a2 |
| 152 | |
| 153 | --2799ac77a72ef7b8a0281945806b9f9a28f7681145aa8e91b052d599b2dd |
| 154 | Content-Disposition: file; filename="test%2Fbaz%2Ff" |
| 155 | Content-Type: application/octet-stream |
| 156 | |
| 157 | |
| 158 | 4 |
| 159 | foo |
| 160 | |
| 161 | 44 |
| 162 | |
| 163 | --2799ac77a72ef7b8a0281945806b9f9a28f7681145aa8e91b052d599b2dd-- |
| 164 | |
| 165 | 9e |
| 166 | |
| 167 | --acdb172fe12f25e8ffae9981ce6f4580abdefb0cae3ceebe464d802866be |
| 168 | Content-Disposition: file; filename="test%2Ffoo" |
| 169 | Content-Type: application/octet-stream |
| 170 | |
| 171 | |
| 172 | 4 |
| 173 | foo |
| 174 | |
| 175 | 44 |
| 176 | |
| 177 | --acdb172fe12f25e8ffae9981ce6f4580abdefb0cae3ceebe464d802866be-- |
| 178 | |
| 179 | 44 |
| 180 | |
| 181 | --2186ef15d8f2c4f100af72d6d345afe36a4d17ef11264ec5b8ec4436447f-- |
| 182 | |
| 183 | 0 |
| 184 | |
| 185 | ``` |
| 186 | |
| 187 | Which produces: http://gateway.ipfs.io/ipfs/QmNtpA5TBNqHrKf3cLQ1AiUKXiE4JmUodbG5gXrajg8wdv |
| 188 |