| 1 | # Developer Guide |
| 2 | |
| 3 | By the end of this guide, you will be able to: |
| 4 | |
| 5 | - Build Kubo from source |
| 6 | - Run the test suites |
| 7 | - Make and verify code changes |
| 8 | |
| 9 | This guide covers the local development workflow. For user documentation, see [docs.ipfs.tech](https://docs.ipfs.tech/). |
| 10 | |
| 11 | ## Table of Contents |
| 12 | |
| 13 | - [Prerequisites](#prerequisites) |
| 14 | - [Quick Start](#quick-start) |
| 15 | - [Building](#building) |
| 16 | - [Running Tests](#running-tests) |
| 17 | - [Running the Linter](#running-the-linter) |
| 18 | - [Common Development Tasks](#common-development-tasks) |
| 19 | - [Code Organization](#code-organization) |
| 20 | - [Architecture](#architecture) |
| 21 | - [Troubleshooting](#troubleshooting) |
| 22 | - [Development Dependencies](#development-dependencies) |
| 23 | - [Further Reading](#further-reading) |
| 24 | |
| 25 | ## Prerequisites |
| 26 | |
| 27 | Before you begin, ensure you have: |
| 28 | |
| 29 | - **Go** - see `go.mod` for the minimum required version |
| 30 | - **Git** |
| 31 | - **GNU Make** |
| 32 | - **GCC** (optional) - required for CGO (Go's C interop); without it, build with `CGO_ENABLED=0` |
| 33 | |
| 34 | ## Quick Start |
| 35 | |
| 36 | ```bash |
| 37 | git clone https://github.com/ipfs/kubo.git |
| 38 | cd kubo |
| 39 | make build |
| 40 | ./cmd/ipfs/ipfs version |
| 41 | ``` |
| 42 | |
| 43 | You should see output like: |
| 44 | |
| 45 | ``` |
| 46 | ipfs version 0.34.0-dev |
| 47 | ``` |
| 48 | |
| 49 | The binary is built to `cmd/ipfs/ipfs`. To install it system-wide: |
| 50 | |
| 51 | ```bash |
| 52 | make install |
| 53 | ``` |
| 54 | |
| 55 | This installs the binary to `$GOPATH/bin`. |
| 56 | |
| 57 | ## Building |
| 58 | |
| 59 | | Command | Description | |
| 60 | |---------|-------------| |
| 61 | | `make build` | build the `ipfs` binary to `cmd/ipfs/ipfs` | |
| 62 | | `make install` | install to `$GOPATH/bin` | |
| 63 | | `make nofuse` | build without FUSE support | |
| 64 | | `make build CGO_ENABLED=0` | build without CGO (no C compiler needed) | |
| 65 | |
| 66 | For Windows-specific instructions, see [windows.md](windows.md). |
| 67 | |
| 68 | ## Running Tests |
| 69 | |
| 70 | Kubo has two types of tests: |
| 71 | |
| 72 | - **Unit tests** - test individual packages in isolation. Fast and don't require a running daemon. |
| 73 | - **End-to-end tests** - spawn real `ipfs` nodes, run actual CLI commands, and test the full system. Slower but catch integration issues. |
| 74 | |
| 75 | Note that `go test ./...` runs both unit and end-to-end tests. Use `make test` to run all tests. CI runs unit and end-to-end tests in separate jobs for faster feedback. |
| 76 | |
| 77 | <!-- TODO: uncomment when https://github.com/ipfs/kubo/pull/11113 is merged |
| 78 | | Command | What it runs | |
| 79 | |---------|--------------| |
| 80 | | `make test_unit` | unit tests only (excludes `test/cli`) | |
| 81 | | `make test_cli` | CLI end-to-end tests only (requires `make build` first) | |
| 82 | | `make test_sharness` | sharness end-to-end tests only | |
| 83 | | `make test` | all tests (unit + CLI + sharness) | |
| 84 | --> |
| 85 | |
| 86 | For end-to-end tests, Kubo has two suites: |
| 87 | |
| 88 | - **`test/cli`** - modern Go-based test harness that spawns real `ipfs` nodes and runs actual CLI commands. All new tests should be added here. |
| 89 | - **`test/sharness`** - legacy bash-based tests. We are slowly migrating these to `test/cli`. |
| 90 | |
| 91 | When modifying tests: cosmetic changes to `test/sharness` are fine, but if significant rewrites are needed, remove the outdated sharness test and add a modern one to `test/cli` instead. |
| 92 | |
| 93 | ### Before Running Tests |
| 94 | |
| 95 | **Environment requirements**: some legacy tests expect default ports (8080, 5001, 4001) to be free and no mDNS (local network discovery) Kubo service on the LAN. Tests may fail if you have a local Kubo instance running. Before running the full test suite, stop any running `ipfs daemon`. |
| 96 | |
| 97 | Two critical setup steps: |
| 98 | |
| 99 | 1. **Rebuild after code changes**: if you modify any `.go` files outside of `test/`, you must run `make build` before running integration tests. |
| 100 | |
| 101 | 2. **Set environment variables**: integration tests use the `ipfs` binary from `PATH` and need an isolated `IPFS_PATH`. Run these commands from the repository root: |
| 102 | |
| 103 | ```bash |
| 104 | export PATH="$PWD/cmd/ipfs:$PATH" |
| 105 | export IPFS_PATH="$(mktemp -d)" |
| 106 | ``` |
| 107 | |
| 108 | ### Unit Tests |
| 109 | |
| 110 | ```bash |
| 111 | go test ./... |
| 112 | ``` |
| 113 | |
| 114 | ### CLI Integration Tests (`test/cli`) |
| 115 | |
| 116 | These are Go-based integration tests that invoke the `ipfs` CLI. |
| 117 | |
| 118 | Instead of running the entire test suite, you can run a specific test to get faster feedback during development. |
| 119 | |
| 120 | Run a specific test (recommended during development): |
| 121 | |
| 122 | ```bash |
| 123 | go test ./test/cli/... -run TestAdd -v |
| 124 | ``` |
| 125 | |
| 126 | Run all CLI tests: |
| 127 | |
| 128 | ```bash |
| 129 | go test ./test/cli/... |
| 130 | ``` |
| 131 | |
| 132 | Run a specific test: |
| 133 | |
| 134 | ```bash |
| 135 | go test ./test/cli/... -run TestAdd |
| 136 | ``` |
| 137 | |
| 138 | Run with verbose output: |
| 139 | |
| 140 | ```bash |
| 141 | go test ./test/cli/... -v |
| 142 | ``` |
| 143 | |
| 144 | **Common error**: "version (16) is lower than repos (17)" means your `PATH` points to an old binary. Check `which ipfs` and rebuild with `make build`. |
| 145 | |
| 146 | ### Sharness Tests (`test/sharness`) |
| 147 | |
| 148 | Shell-based integration tests using [sharness](https://github.com/chriscool/sharness) (a portable shell testing framework). |
| 149 | |
| 150 | ```bash |
| 151 | cd test/sharness |
| 152 | ``` |
| 153 | |
| 154 | Run a specific test: |
| 155 | |
| 156 | ```bash |
| 157 | timeout 60s ./t0080-repo.sh |
| 158 | ``` |
| 159 | |
| 160 | Run with verbose output (this disables automatic cleanup): |
| 161 | |
| 162 | ```bash |
| 163 | ./t0080-repo.sh -v |
| 164 | ``` |
| 165 | |
| 166 | **Cleanup**: the `-v` flag disables automatic cleanup. Before re-running tests, kill any dangling `ipfs daemon` processes: |
| 167 | |
| 168 | ```bash |
| 169 | pkill -f "ipfs daemon" |
| 170 | ``` |
| 171 | |
| 172 | ### Full Test Suite |
| 173 | |
| 174 | ```bash |
| 175 | make test # run all tests |
| 176 | make test_short # run shorter test suite |
| 177 | ``` |
| 178 | |
| 179 | ## Running the Linter |
| 180 | |
| 181 | Run the linter using the Makefile target (not `golangci-lint` directly): |
| 182 | |
| 183 | ```bash |
| 184 | make -O test_go_lint |
| 185 | ``` |
| 186 | |
| 187 | ## Common Development Tasks |
| 188 | |
| 189 | ### Modifying CLI Commands |
| 190 | |
| 191 | After editing help text in `core/commands/`, verify the output width: |
| 192 | |
| 193 | ```bash |
| 194 | go test ./test/cli/... -run TestCommandDocsWidth |
| 195 | ``` |
| 196 | |
| 197 | ### Updating Dependencies |
| 198 | |
| 199 | Use the Makefile target (not `go mod tidy` directly): |
| 200 | |
| 201 | ```bash |
| 202 | make mod_tidy |
| 203 | ``` |
| 204 | |
| 205 | ### Editing the Changelog |
| 206 | |
| 207 | When modifying `docs/changelogs/`: |
| 208 | |
| 209 | - update the Table of Contents when adding sections |
| 210 | - add user-facing changes to the Highlights section (the Changelog section is auto-generated) |
| 211 | |
| 212 | ### Running the Daemon |
| 213 | |
| 214 | Always run the daemon with a timeout or shut it down promptly. |
| 215 | |
| 216 | With timeout: |
| 217 | |
| 218 | ```bash |
| 219 | timeout 60s ipfs daemon |
| 220 | ``` |
| 221 | |
| 222 | Or shut down via API: |
| 223 | |
| 224 | ```bash |
| 225 | ipfs shutdown |
| 226 | ``` |
| 227 | |
| 228 | For multi-step experiments, store `IPFS_PATH` in a file to ensure consistency. |
| 229 | |
| 230 | ## Code Organization |
| 231 | |
| 232 | | Directory | Description | |
| 233 | |-----------|-------------| |
| 234 | | `cmd/ipfs/` | CLI entry point and binary | |
| 235 | | `core/` | core IPFS node implementation | |
| 236 | | `core/commands/` | CLI command definitions | |
| 237 | | `core/coreapi/` | Go API implementation | |
| 238 | | `client/rpc/` | HTTP RPC client | |
| 239 | | `plugin/` | plugin system | |
| 240 | | `repo/` | repository management | |
| 241 | | `test/cli/` | Go-based CLI integration tests | |
| 242 | | `test/sharness/` | legacy shell-based integration tests | |
| 243 | | `docs/` | documentation | |
| 244 | |
| 245 | Key external dependencies: |
| 246 | |
| 247 | - [go-libp2p](https://github.com/libp2p/go-libp2p) - networking stack |
| 248 | - [go-libp2p-kad-dht](https://github.com/libp2p/go-libp2p-kad-dht) - distributed hash table |
| 249 | - [boxo](https://github.com/ipfs/boxo) - IPFS SDK (including Bitswap, the data exchange engine) |
| 250 | |
| 251 | For a deep dive into how code flows through Kubo, see [The `Add` command demystified](add-code-flow.md). |
| 252 | |
| 253 | ## Architecture |
| 254 | |
| 255 | **Map of Implemented Subsystems** ([editable source](https://docs.google.com/drawings/d/1OVpBT2q-NtSJqlPX3buvjYhOnWfdzb85YEsM_njesME/edit)): |
| 256 | |
| 257 | <img src="https://docs.google.com/drawings/d/e/2PACX-1vS_n1FvSu6mdmSirkBrIIEib2gqhgtatD9awaP2_WdrGN4zTNeg620XQd9P95WT-IvognSxIIdCM5uE/pub?w=1446&h=1036"> |
| 258 | |
| 259 | **CLI, HTTP-API, Core Diagram**: |
| 260 | |
| 261 |  |
| 262 | |
| 263 | ## Troubleshooting |
| 264 | |
| 265 | ### "version (N) is lower than repos (M)" Error |
| 266 | |
| 267 | This means the `ipfs` binary in your `PATH` is older than expected. |
| 268 | |
| 269 | Check which binary is being used: |
| 270 | |
| 271 | ```bash |
| 272 | which ipfs |
| 273 | ``` |
| 274 | |
| 275 | Rebuild and verify PATH: |
| 276 | |
| 277 | ```bash |
| 278 | make build |
| 279 | export PATH="$PWD/cmd/ipfs:$PATH" |
| 280 | ./cmd/ipfs/ipfs version |
| 281 | ``` |
| 282 | |
| 283 | ### FUSE Issues |
| 284 | |
| 285 | If you don't need FUSE support, build without it: |
| 286 | |
| 287 | ```bash |
| 288 | make nofuse |
| 289 | ``` |
| 290 | |
| 291 | Or set the `TEST_FUSE=0` environment variable when running tests. |
| 292 | |
| 293 | ### Build Fails with "No such file: stdlib.h" |
| 294 | |
| 295 | You're missing a C compiler. Either install GCC or build without CGO: |
| 296 | |
| 297 | ```bash |
| 298 | make build CGO_ENABLED=0 |
| 299 | ``` |
| 300 | |
| 301 | ## Development Dependencies |
| 302 | |
| 303 | If you make changes to the protocol buffers, you will need to install the [protoc compiler](https://github.com/google/protobuf). |
| 304 | |
| 305 | ## Further Reading |
| 306 | |
| 307 | - [The `Add` command demystified](add-code-flow.md) - deep dive into code flow |
| 308 | - [Configuration reference](config.md) |
| 309 | - [Performance debugging](debug-guide.md) |
| 310 | - [Experimental features](experimental-features.md) |
| 311 | - [Release process](releases.md) |
| 312 | - [Contributing guidelines](https://github.com/ipfs/community/blob/master/CONTRIBUTING.md) |
| 313 | |
| 314 | ## Source Code |
| 315 | |
| 316 | The complete source code is at [github.com/ipfs/kubo](https://github.com/ipfs/kubo). |