fix(ci): parallelize gotest, cleanup output, flakiness (#11113)
* ci: parallelize gotest by separating test/cli into own job split the Go Test workflow into two parallel jobs: - `unit-tests`: runs unit tests (excluding test/cli) - `cli-tests`: runs test/cli end-to-end tests test/cli takes ~3 minutes (~50% of total gotest time), so running it in parallel should reduce wall-clock CI time by ~1.5-2.5 minutes. both jobs produce JUnit XML and HTML reports for consistent debugging. * ci(gotest): reduce noise on test timeout panics add GOTRACEBACK=single to show only one goroutine stack instead of all when a test timeout panic occurs. this makes CI output much cleaner when tests hang. * fix(ci): prevent stderr from corrupting test JSON output - remove 2>&1 which mixed "go: downloading" stderr messages into JSON - add JSON validation before parsing - print failed test names for easier debugging * ci(gotest): use gotestsum for human-readable test output - replace per-package coverage loop with single gotestsum invocation - both unit-tests and cli-tests now show human-readable output - simplified coverage collection (single -coverprofile, no gocovmerge) - clarified step names to indicate they run tests * ci: fix codecov uploads by adding token - add CODECOV_TOKEN to gotest.yml and sharness.yml - update codecov-action to v5.5.2 - add fail_ci_if_error: false for robustness codecov stopped receiving coverage data ~1 year ago when they started requiring tokens for public repos * refactor(make): add test_unit and test_cli targets - add `make test_unit` for unit tests with coverage (used by CI) - add `make test_cli` for CLI integration tests (used by CI) - only disable colors when CI env var is set (local dev gets colors) - remove legacy targets: test_go_test, test_go_short, test_go_race, test_go_expensive - update gotest.yml to use make targets instead of inline commands - add test artifacts to .gitignore * fix(ci): move client/rpc tests to cli-tests job client/rpc tests use test/cli/harness which requires the ipfs binary. Move them from test_unit to test_cli where the binary is built. also: - update gotestsum to v1.13.0 - simplify workflow step names * fix(ci): use build tags when listing test packages go list needs build tags to properly exclude packages like fuse/mfs when running with TEST_FUSE=0 (nofuse tag). * fix(ci): move test/integration to cli-tests job test/integration tests need the ipfs binary, move them from test_unit to test_cli. * fix(test): fix flaky kubo-as-a-library and GetClosestPeers tests kubo-as-a-library: use `Bootstrap()` instead of raw `Swarm().Connect()` to fix race condition between swarm connection and bitswap peer discovery. `Bootstrap()` properly integrates peers into the routing system, ensuring bitswap learns about connected peers synchronously. GetClosestPeers: simplify retry logic using `EventuallyWithT` with 10-minute timeout. tests all 4 routing types (`auto`, `autoclient`, `dht`, `dhtclient`) against real bootstrap peers with patient polling. * fix(example): use bidirectional Swarm().Connect() for reliable bitswap - connect nodes bidirectionally (A→B and B→A) to simulate mutual peering - mutual peering protects connection from resource manager culling - use port 0 for random available ports (avoids CI conflicts) - enable LoopbackAddressesOnLanDHT for local testing - move retry logic to test file using require.Eventually * fix(ci): add test_examples target and parallel example-tests job - add `make test_examples` target to mk/golang.mk for consistency with test_unit/test_cli - move example tests to separate parallel CI job (example-tests) - example: use Bootstrap() with autoconf.FallbackBootstrapPeers for reliable bitswap - example: increase context timeout to 10 minutes - test: add 60s per-request timeout to GetClosestPeers (server has 30s routing timeout) - test: reduce EventuallyWithT to 3 minutes (locally passes in under 1 minute) * fix(ci): improve test targets, exclusion patterns, and artifact naming - define COVERPKG_EXCLUDE and UNIT_EXCLUDE as documented variables - use grep -vE with single regex instead of multiple grep -v calls - add mkdir -p before rm to ensure directories exist - add DEPS_GO dependency to test_cli target - make CLI test timeout configurable via TEST_CLI_TIMEOUT (default 10m) - fix test_examples cleanup on failure using subshell - reduce GetClosestPeers test wait time from 3m to 2m - rename artifacts to match job names: unit-tests-{junit,html}, cli-tests-{junit,html} - update cli-tests upload-artifact from v5 to v6 * fix(ci): fix unit test exclusion and speed up example test - fix UNIT_EXCLUDE regex to match client/rpc at end of path - remove public bootstrap peers from example (only connect to nodeA) - example test now runs in ~3s instead of timing out * fix(test): fix flaky TestAddMultipleGCLive race condition added time.Sleep after spawning GC goroutines to ensure they reach GCLock() before the test proceeds. without this, the adder's maybePauseForGC() might check GCRequested() before GC has even requested the lock, causing the lock to not be released and GC to block indefinitely. this matches the existing pattern in TestAddGCLive which already had this sleep. also replaced context.Background() with t.Context() in both TestAddMultipleGCLive and TestAddGCLive for proper test lifecycle management. * fix(example): use test harness settings for reliable CI the kubo-as-a-library example was flaky on CI. applied test-harness-like settings that match what transports_test.go uses: - TCP-only on 127.0.0.1 with random port (no QUIC/UDP) - explicitly disable non-TCP transports (QUIC, Relay, WebTransport, etc) - use NilRouterOption (no routing) since we connect peers directly - bitswap works with directly connected peers without DHT lookups - 2-minute context timeout - streaming output in test for debugging
Marcin Rataj committed
Jan 8, 2026 at 05:07 UTC
1301710a911e146c32456f75fc509b2663ec753e
13 files changed
+263
-183
.github/workflows/gotest.yml
+93
-31
@@ -14,11 +14,13 @@ concurrency:
14
cancel-in-progress: true
15
16
jobs:
17
- go-test:
17
+ # Unit tests with coverage collection (uploaded to Codecov)
18
+ unit-tests:
19
if: github.repository == 'ipfs/kubo' || github.event_name == 'workflow_dispatch'
20
runs-on: ${{ fromJSON(github.repository == 'ipfs/kubo' && '["self-hosted", "linux", "x64", "2xlarge"]' || '"ubuntu-latest"') }}
20
- timeout-minutes: 20
21
+ timeout-minutes: 15
22
env:
23
+ GOTRACEBACK: single # reduce noise on test timeout panics
24
TEST_DOCKER: 0
25
TEST_FUSE: 0
26
TEST_VERBOSE: 1
@@ -36,12 +38,9 @@ jobs:
38
go-version-file: 'go.mod'
39
- name: Install missing tools
40
run: sudo apt update && sudo apt install -y zsh
39
- - name: 👉️ If this step failed, go to «Summary» (top left) → inspect the «Failures/Errors» table
40
- env:
41
- # increasing parallelism beyond 2 doesn't speed up the tests much
42
- PARALLEL: 2
41
+ - name: Run unit tests
42
run: |
44
- make -j "$PARALLEL" test/unit/gotest.junit.xml &&
43
+ make test_unit &&
44
[[ ! $(jq -s -c 'map(select(.Action == "fail")) | .[]' test/unit/gotest.json) ]]
45
- name: Upload coverage to Codecov
46
uses: codecov/codecov-action@671740ac38dd9b0130fbe1cec585b89eea48d3de # v5.5.2
@@ -49,28 +48,8 @@ jobs:
48
with:
49
name: unittests
50
files: coverage/unit_tests.coverprofile
52
- - name: Test kubo-as-a-library example
53
- run: |
54
- # we want to first test with the kubo version in the go.mod file
55
- go test -v ./...
56
-
57
- # we also want to test the examples against the current version of kubo
58
- # however, that version might be in a fork so we need to replace the dependency
59
-
60
- # backup the go.mod and go.sum files to restore them after we run the tests
61
- cp go.mod go.mod.bak
62
- cp go.sum go.sum.bak
63
-
64
- # make sure the examples run against the current version of kubo
65
- go mod edit -replace github.com/ipfs/kubo=./../../..
66
- go mod tidy
67
-
68
- go test -v ./...
69
-
70
- # restore the go.mod and go.sum files to their original state
71
- mv go.mod.bak go.mod
72
- mv go.sum.bak go.sum
73
- working-directory: docs/examples/kubo-as-a-library
51
+ token: ${{ secrets.CODECOV_TOKEN }}
52
+ fail_ci_if_error: false
53
- name: Create a proper JUnit XML report
54
uses: ipdxco/gotest-json-to-junit-xml@v1
55
with:
@@ -80,7 +59,7 @@ jobs:
59
- name: Archive the JUnit XML report
60
uses: actions/upload-artifact@v6
61
with:
83
- name: unit
62
+ name: unit-tests-junit
63
path: test/unit/gotest.junit.xml
64
if: failure() || success()
65
- name: Create a HTML report
@@ -93,7 +72,7 @@ jobs:
72
- name: Archive the HTML report
73
uses: actions/upload-artifact@v6
74
with:
96
- name: html
75
+ name: unit-tests-html
76
path: test/unit/gotest.html
77
if: failure() || success()
78
- name: Create a Markdown report
@@ -106,3 +85,86 @@ jobs:
85
- name: Set the summary
86
run: cat test/unit/gotest.md >> $GITHUB_STEP_SUMMARY
87
if: failure() || success()
88
+
89
+ # End-to-end integration/regression tests from test/cli
90
+ # (Go-based replacement for legacy test/sharness shell scripts)
91
+ cli-tests:
92
+ if: github.repository == 'ipfs/kubo' || github.event_name == 'workflow_dispatch'
93
+ runs-on: ${{ fromJSON(github.repository == 'ipfs/kubo' && '["self-hosted", "linux", "x64", "2xlarge"]' || '"ubuntu-latest"') }}
94
+ timeout-minutes: 15
95
+ env:
96
+ GOTRACEBACK: single # reduce noise on test timeout panics
97
+ TEST_VERBOSE: 1
98
+ GIT_PAGER: cat
99
+ IPFS_CHECK_RCMGR_DEFAULTS: 1
100
+ defaults:
101
+ run:
102
+ shell: bash
103
+ steps:
104
+ - name: Check out Kubo
105
+ uses: actions/checkout@v6
106
+ - name: Set up Go
107
+ uses: actions/setup-go@v6
108
+ with:
109
+ go-version-file: 'go.mod'
110
+ - name: Install missing tools
111
+ run: sudo apt update && sudo apt install -y zsh
112
+ - name: Run CLI tests
113
+ env:
114
+ IPFS_PATH: ${{ runner.temp }}/ipfs-test
115
+ run: make test_cli
116
+ - name: Create JUnit XML report
117
+ uses: ipdxco/gotest-json-to-junit-xml@v1
118
+ with:
119
+ input: test/cli/cli-tests.json
120
+ output: test/cli/cli-tests.junit.xml
121
+ if: failure() || success()
122
+ - name: Archive JUnit XML report
123
+ uses: actions/upload-artifact@v6
124
+ with:
125
+ name: cli-tests-junit
126
+ path: test/cli/cli-tests.junit.xml
127
+ if: failure() || success()
128
+ - name: Create HTML report
129
+ uses: ipdxco/junit-xml-to-html@v1
130
+ with:
131
+ mode: no-frames
132
+ input: test/cli/cli-tests.junit.xml
133
+ output: test/cli/cli-tests.html
134
+ if: failure() || success()
135
+ - name: Archive HTML report
136
+ uses: actions/upload-artifact@v6
137
+ with:
138
+ name: cli-tests-html
139
+ path: test/cli/cli-tests.html
140
+ if: failure() || success()
141
+ - name: Create Markdown report
142
+ uses: ipdxco/junit-xml-to-html@v1
143
+ with:
144
+ mode: summary
145
+ input: test/cli/cli-tests.junit.xml
146
+ output: test/cli/cli-tests.md
147
+ if: failure() || success()
148
+ - name: Set summary
149
+ run: cat test/cli/cli-tests.md >> $GITHUB_STEP_SUMMARY
150
+ if: failure() || success()
151
+
152
+ # Example tests (kubo-as-a-library)
153
+ example-tests:
154
+ if: github.repository == 'ipfs/kubo' || github.event_name == 'workflow_dispatch'
155
+ runs-on: ${{ fromJSON(github.repository == 'ipfs/kubo' && '["self-hosted", "linux", "x64", "2xlarge"]' || '"ubuntu-latest"') }}
156
+ timeout-minutes: 5
157
+ env:
158
+ GOTRACEBACK: single
159
+ defaults:
160
+ run:
161
+ shell: bash
162
+ steps:
163
+ - name: Check out Kubo
164
+ uses: actions/checkout@v6
165
+ - name: Set up Go
166
+ uses: actions/setup-go@v6
167
+ with:
168
+ go-version-file: 'go.mod'
169
+ - name: Run example tests
170
+ run: make test_examples
.github/workflows/sharness.yml
+2
@@ -60,6 +60,8 @@ jobs:
60
with:
61
name: sharness
62
files: kubo/coverage/sharness_tests.coverprofile
63
+ token: ${{ secrets.CODECOV_TOKEN }}
64
+ fail_ci_if_error: false
65
- name: Aggregate results
66
run: find kubo/test/sharness/test-results -name 't*-*.sh.*.counts' | kubo/test/sharness/lib/sharness/aggregate-results.sh > kubo/test/sharness/test-results/summary.txt
67
- name: 👉️ If this step failed, go to «Summary» (top left) → «HTML Report» → inspect the «Failures» column
.gitignore
+5
@@ -28,6 +28,11 @@ go-ipfs-source.tar.gz
28
docs/examples/go-ipfs-as-a-library/example-folder/Qm*
29
/test/sharness/t0054-dag-car-import-export-data/*.car
30
31
+# test artifacts from make test_unit / test_cli
32
+/test/unit/gotest.json
33
+/test/unit/gotest.junit.xml
34
+/test/cli/cli-tests.json
35
+
36
# ignore build output from snapcraft
37
/ipfs_*.snap
38
/parts
Rules.mk
+7
-8
@@ -134,15 +134,14 @@ help:
134
@echo ''
135
@echo 'TESTING TARGETS:'
136
@echo ''
137
- @echo ' test - Run all tests'
138
- @echo ' test_short - Run short go tests and short sharness tests'
139
- @echo ' test_go_short - Run short go tests'
140
- @echo ' test_go_test - Run all go tests'
137
+ @echo ' test - Run all tests (test_go_fmt, test_unit, test_cli, test_sharness)'
138
+ @echo ' test_short - Run fast tests (test_go_fmt, test_unit)'
139
+ @echo ' test_unit - Run unit tests with coverage (excludes test/cli)'
140
+ @echo ' test_cli - Run CLI integration tests (requires built binary)'
141
+ @echo ' test_go_fmt - Check Go source formatting'
142
@echo ' test_go_build - Build kubo for all platforms from .github/build-platforms.yml'
142
- @echo ' test_go_expensive - Run all go tests and build all platforms'
143
- @echo ' test_go_race - Run go tests with the race detector enabled'
144
- @echo ' test_go_lint - Run the `golangci-lint` vetting tool'
143
+ @echo ' test_go_lint - Run golangci-lint'
144
@echo ' test_sharness - Run sharness tests'
146
- @echo ' coverage - Collects coverage info from unit tests and sharness'
145
+ @echo ' coverage - Collect coverage info from unit tests and sharness'
146
@echo
147
.PHONY: help
core/coreunix/add_test.go
+17
-12
@@ -30,6 +30,7 @@ import (
30
const testPeerID = "QmTFauExutTsy4XP6JbMFcw2Wa9645HJt2bTqL6qYDCKfe"
31
32
func TestAddMultipleGCLive(t *testing.T) {
33
+ ctx := t.Context()
34
r := &repo.Mock{
35
C: config.Config{
36
Identity: config.Identity{
@@ -38,13 +39,13 @@ func TestAddMultipleGCLive(t *testing.T) {
39
},
40
D: syncds.MutexWrap(datastore.NewMapDatastore()),
41
}
41
- node, err := core.NewNode(context.Background(), &core.BuildCfg{Repo: r})
42
+ node, err := core.NewNode(ctx, &core.BuildCfg{Repo: r})
43
if err != nil {
44
t.Fatal(err)
45
}
46
47
out := make(chan interface{}, 10)
47
- adder, err := NewAdder(context.Background(), node.Pinning, node.Blockstore, node.DAG)
48
+ adder, err := NewAdder(ctx, node.Pinning, node.Blockstore, node.DAG)
49
if err != nil {
50
t.Fatal(err)
51
}
@@ -67,7 +68,7 @@ func TestAddMultipleGCLive(t *testing.T) {
68
69
go func() {
70
defer close(out)
70
- _, _ = adder.AddAllAndPin(context.Background(), slf)
71
+ _, _ = adder.AddAllAndPin(ctx, slf)
72
// Ignore errors for clarity - the real bug would be gc'ing files while adding them, not this resultant error
73
}()
74
@@ -80,9 +81,12 @@ func TestAddMultipleGCLive(t *testing.T) {
81
gc1started := make(chan struct{})
82
go func() {
83
defer close(gc1started)
83
- gc1out = gc.GC(context.Background(), node.Blockstore, node.Repo.Datastore(), node.Pinning, nil)
84
+ gc1out = gc.GC(ctx, node.Blockstore, node.Repo.Datastore(), node.Pinning, nil)
85
}()
86
87
+ // Give GC goroutine time to reach GCLock (will block there waiting for adder)
88
+ time.Sleep(time.Millisecond * 100)
89
+
90
// GC shouldn't get the lock until after the file is completely added
91
select {
92
case <-gc1started:
@@ -119,9 +123,12 @@ func TestAddMultipleGCLive(t *testing.T) {
123
gc2started := make(chan struct{})
124
go func() {
125
defer close(gc2started)
122
- gc2out = gc.GC(context.Background(), node.Blockstore, node.Repo.Datastore(), node.Pinning, nil)
126
+ gc2out = gc.GC(ctx, node.Blockstore, node.Repo.Datastore(), node.Pinning, nil)
127
}()
128
129
+ // Give GC goroutine time to reach GCLock
130
+ time.Sleep(time.Millisecond * 100)
131
+
132
select {
133
case <-gc2started:
134
t.Fatal("gc shouldn't have started yet")
@@ -155,6 +162,7 @@ func TestAddMultipleGCLive(t *testing.T) {
162
}
163
164
func TestAddGCLive(t *testing.T) {
165
+ ctx := t.Context()
166
r := &repo.Mock{
167
C: config.Config{
168
Identity: config.Identity{
@@ -163,13 +171,13 @@ func TestAddGCLive(t *testing.T) {
171
},
172
D: syncds.MutexWrap(datastore.NewMapDatastore()),
173
}
166
- node, err := core.NewNode(context.Background(), &core.BuildCfg{Repo: r})
174
+ node, err := core.NewNode(ctx, &core.BuildCfg{Repo: r})
175
if err != nil {
176
t.Fatal(err)
177
}
178
179
out := make(chan interface{})
172
- adder, err := NewAdder(context.Background(), node.Pinning, node.Blockstore, node.DAG)
180
+ adder, err := NewAdder(ctx, node.Pinning, node.Blockstore, node.DAG)
181
if err != nil {
182
t.Fatal(err)
183
}
@@ -193,7 +201,7 @@ func TestAddGCLive(t *testing.T) {
201
go func() {
202
defer close(addDone)
203
defer close(out)
196
- _, err := adder.AddAllAndPin(context.Background(), slf)
204
+ _, err := adder.AddAllAndPin(ctx, slf)
205
if err != nil {
206
t.Error(err)
207
}
@@ -211,7 +219,7 @@ func TestAddGCLive(t *testing.T) {
219
gcstarted := make(chan struct{})
220
go func() {
221
defer close(gcstarted)
214
- gcout = gc.GC(context.Background(), node.Blockstore, node.Repo.Datastore(), node.Pinning, nil)
222
+ gcout = gc.GC(ctx, node.Blockstore, node.Repo.Datastore(), node.Pinning, nil)
223
}()
224
225
// gc shouldn't start until we let the add finish its current file.
@@ -255,9 +263,6 @@ func TestAddGCLive(t *testing.T) {
263
last = c
264
}
265
258
- ctx, cancel := context.WithTimeout(context.Background(), time.Second*5)
259
- defer cancel()
260
-
266
set := cid.NewSet()
267
err = dag.Walk(ctx, dag.GetLinksWithDAG(node.DAG), last, set.Visit)
268
if err != nil {
coverage/Rules.mk
+4
-23
@@ -3,33 +3,14 @@ include mk/header.mk
3
GOCC ?= go
4
5
$(d)/coverage_deps: $$(DEPS_GO) cmd/ipfs/ipfs
6
- rm -rf $(@D)/unitcover && mkdir $(@D)/unitcover
6
rm -rf $(@D)/sharnesscover && mkdir $(@D)/sharnesscover
7
9
-ifneq ($(IPFS_SKIP_COVER_BINS),1)
10
-$(d)/coverage_deps: test/bin/gocovmerge
11
-endif
12
-
8
.PHONY: $(d)/coverage_deps
9
15
-# unit tests coverage
16
-UTESTS_$(d) := $(shell $(GOCC) list -f '{{if (or (len .TestGoFiles) (len .XTestGoFiles))}}{{.ImportPath}}{{end}}' $(go-flags-with-tags) ./... | grep -v go-ipfs/vendor | grep -v go-ipfs/Godeps)
17
-
18
-UCOVER_$(d) := $(addsuffix .coverprofile,$(addprefix $(d)/unitcover/, $(subst /,_,$(UTESTS_$(d)))))
19
-
20
-$(UCOVER_$(d)): $(d)/coverage_deps ALWAYS
21
- $(eval TMP_PKG := $(subst _,/,$(basename $(@F))))
22
- $(eval TMP_DEPS := $(shell $(GOCC) list -f '{{range .Deps}}{{.}} {{end}}' $(go-flags-with-tags) $(TMP_PKG) | sed 's/ /\n/g' | grep ipfs/go-ipfs) $(TMP_PKG))
23
- $(eval TMP_DEPS_LIST := $(call join-with,$(comma),$(TMP_DEPS)))
24
- $(GOCC) test $(go-flags-with-tags) $(GOTFLAGS) -v -covermode=atomic -json -coverpkg=$(TMP_DEPS_LIST) -coverprofile=$@ $(TMP_PKG) | tee -a test/unit/gotest.json
25
-
26
-
27
-$(d)/unit_tests.coverprofile: $(UCOVER_$(d))
28
- gocovmerge $^ > $@
29
-
30
-TGTS_$(d) := $(d)/unit_tests.coverprofile
10
+# unit tests coverage is now produced by test_unit target in mk/golang.mk
11
+# (outputs coverage/unit_tests.coverprofile and test/unit/gotest.json)
12
32
-.PHONY: $(d)/unit_tests.coverprofile
13
+TGTS_$(d) :=
14
15
# sharness tests coverage
16
$(d)/ipfs: GOTAGS += testrunmain
@@ -46,7 +27,7 @@ endif
27
export IPFS_COVER_DIR:= $(realpath $(d))/sharnesscover/
28
29
$(d)/sharness_tests.coverprofile: export TEST_PLUGIN=0
49
-$(d)/sharness_tests.coverprofile: $(d)/ipfs cmd/ipfs/ipfs-test-cover $(d)/coverage_deps test_sharness
30
+$(d)/sharness_tests.coverprofile: $(d)/ipfs cmd/ipfs/ipfs-test-cover $(d)/coverage_deps test/bin/gocovmerge test_sharness
31
(cd $(@D)/sharnesscover && find . -type f | gocovmerge -list -) > $@
32
33
docs/examples/kubo-as-a-library/main.go
+40
-33
@@ -47,7 +47,7 @@ func setupPlugins(externalPluginsPath string) error {
47
return nil
48
}
49
50
-func createTempRepo(swarmPort int) (string, error) {
50
+func createTempRepo() (string, error) {
51
repoPath, err := os.MkdirTemp("", "ipfs-shell")
52
if err != nil {
53
return "", fmt.Errorf("failed to get temp dir: %s", err)
@@ -59,15 +59,28 @@ func createTempRepo(swarmPort int) (string, error) {
59
return "", err
60
}
61
62
- // Configure custom ports to avoid conflicts with other IPFS instances.
63
- // This demonstrates how to customize the node's network addresses.
62
+ // Use TCP-only on loopback with random port for reliable local testing.
63
+ // This matches what kubo's test harness uses (test/cli/transports_test.go).
64
+ // QUIC/UDP transports are avoided because they may be throttled on CI.
65
cfg.Addresses.Swarm = []string{
65
- fmt.Sprintf("/ip4/0.0.0.0/tcp/%d", swarmPort),
66
- fmt.Sprintf("/ip4/0.0.0.0/udp/%d/quic-v1", swarmPort),
67
- fmt.Sprintf("/ip4/0.0.0.0/udp/%d/quic-v1/webtransport", swarmPort),
68
- fmt.Sprintf("/ip4/0.0.0.0/udp/%d/webrtc-direct", swarmPort),
66
+ "/ip4/127.0.0.1/tcp/0",
67
}
68
69
+ // Explicitly disable non-TCP transports for reliability.
70
+ cfg.Swarm.Transports.Network.QUIC = config.False
71
+ cfg.Swarm.Transports.Network.Relay = config.False
72
+ cfg.Swarm.Transports.Network.WebTransport = config.False
73
+ cfg.Swarm.Transports.Network.WebRTCDirect = config.False
74
+ cfg.Swarm.Transports.Network.Websocket = config.False
75
+ cfg.AutoTLS.Enabled = config.False
76
+
77
+ // Disable routing - we don't need DHT for direct peer connections.
78
+ // Bitswap works with directly connected peers without needing DHT lookups.
79
+ cfg.Routing.Type = config.NewOptionalString("none")
80
+
81
+ // Disable bootstrap for this example - we manually connect only the peers we need.
82
+ cfg.Bootstrap = []string{}
83
+
84
// When creating the repository, you can define custom settings on the repository, such as enabling experimental
85
// features (See experimental-features.md) or customizing the gateway endpoint.
86
// To do such things, you should modify the variable `cfg`. For example:
@@ -106,10 +119,14 @@ func createNode(ctx context.Context, repoPath string) (*core.IpfsNode, error) {
119
// Construct the node
120
121
nodeOptions := &core.BuildCfg{
109
- Online: true,
110
- Routing: libp2p.DHTOption, // This option sets the node to be a full DHT node (both fetching and storing DHT Records)
111
- // Routing: libp2p.DHTClientOption, // This option sets the node to be a client DHT node (only fetching records)
112
- Repo: repo,
122
+ Online: true,
123
+ // For this example, we use NilRouterOption (no routing) since we connect peers directly.
124
+ // Bitswap works with directly connected peers without needing DHT lookups.
125
+ // In production, you would typically use:
126
+ // Routing: libp2p.DHTOption, // Full DHT node (stores and fetches records)
127
+ // Routing: libp2p.DHTClientOption, // DHT client (only fetches records)
128
+ Routing: libp2p.NilRouterOption,
129
+ Repo: repo,
130
}
131
132
return core.NewNode(ctx, nodeOptions)
@@ -118,8 +135,7 @@ func createNode(ctx context.Context, repoPath string) (*core.IpfsNode, error) {
135
var loadPluginsOnce sync.Once
136
137
// Spawns a node to be used just for this run (i.e. creates a tmp repo).
121
-// The swarmPort parameter specifies the port for libp2p swarm listeners.
122
-func spawnEphemeral(ctx context.Context, swarmPort int) (icore.CoreAPI, *core.IpfsNode, error) {
138
+func spawnEphemeral(ctx context.Context) (icore.CoreAPI, *core.IpfsNode, error) {
139
var onceErr error
140
loadPluginsOnce.Do(func() {
141
onceErr = setupPlugins("")
@@ -129,7 +145,7 @@ func spawnEphemeral(ctx context.Context, swarmPort int) (icore.CoreAPI, *core.Ip
145
}
146
147
// Create a Temporary Repo
132
- repoPath, err := createTempRepo(swarmPort)
148
+ repoPath, err := createTempRepo()
149
if err != nil {
150
return nil, nil, fmt.Errorf("failed to create temp repo: %s", err)
151
}
@@ -207,8 +223,7 @@ func main() {
223
defer cancel()
224
225
// Spawn a local peer using a temporary path, for testing purposes
210
- // Using port 4010 to avoid conflict with default IPFS port 4001
211
- ipfsA, nodeA, err := spawnEphemeral(ctx, 4010)
226
+ ipfsA, nodeA, err := spawnEphemeral(ctx)
227
if err != nil {
228
panic(fmt.Errorf("failed to spawn peer node: %s", err))
229
}
@@ -222,9 +237,8 @@ func main() {
237
fmt.Printf("Added file to peer with CID %s\n", peerCidFile.String())
238
239
// Spawn a node using a temporary path, creating a temporary repo for the run
225
- // Using port 4011 (different from nodeA's port 4010)
240
fmt.Println("Spawning Kubo node on a temporary repo")
227
- ipfsB, _, err := spawnEphemeral(ctx, 4011)
241
+ ipfsB, _, err := spawnEphemeral(ctx)
242
if err != nil {
243
panic(fmt.Errorf("failed to spawn ephemeral node: %s", err))
244
}
@@ -297,11 +311,12 @@ func main() {
311
312
fmt.Printf("Got directory back from IPFS (IPFS path: %s) and wrote it to %s\n", cidDirectory.String(), outputPathDirectory)
313
300
- /// --- Part IV: Getting a file from the IPFS Network
314
+ /// --- Part IV: Getting a file from another IPFS node
315
302
- fmt.Println("\n-- Going to connect to a few nodes in the Network as bootstrappers --")
316
+ fmt.Println("\n-- Connecting to nodeA and fetching content via bitswap --")
317
304
- // Get nodeA's address so we can fetch the file we added to it
318
+ // Get nodeA's actual listening address dynamically.
319
+ // We configured TCP-only on 127.0.0.1 with random port, so this will be a TCP address.
320
peerAddrs, err := ipfsA.Swarm().LocalAddrs(ctx)
321
if err != nil {
322
panic(fmt.Errorf("could not get peer addresses: %s", err))
@@ -309,26 +324,18 @@ func main() {
324
peerMa := peerAddrs[0].String() + "/p2p/" + nodeA.Identity.String()
325
326
bootstrapNodes := []string{
312
- // In production, use autoconf.FallbackBootstrapPeers from boxo/autoconf
313
- // which includes well-known IPFS bootstrap peers like:
327
+ // In production, use real bootstrap peers like:
328
// "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
315
- // "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
316
- // "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
317
-
318
- // You can add custom peers here. For example, another IPFS node:
319
- // "/ip4/192.0.2.1/tcp/4001/p2p/QmYourPeerID...",
320
- // "/ip4/192.0.2.1/udp/4001/quic-v1/p2p/QmYourPeerID...",
321
-
322
- // nodeA's address (the peer we created above that has our test file)
329
+ // For this example, we only connect to nodeA which has our test content.
330
peerMa,
331
}
332
326
- fmt.Println("Connecting to peers...")
333
+ fmt.Println("Connecting to peer...")
334
err = connectToPeers(ctx, ipfsB, bootstrapNodes)
335
if err != nil {
336
panic(fmt.Errorf("failed to connect to peers: %s", err))
337
}
331
- fmt.Println("Connected to peers")
338
+ fmt.Println("Connected to peer")
339
340
exampleCIDStr := peerCidFile.RootCid().String()
341
docs/examples/kubo-as-a-library/main_test.go
+26
-8
@@ -1,21 +1,39 @@
1
package main
2
3
import (
4
+ "bytes"
5
+ "io"
6
+ "os"
7
"os/exec"
8
"strings"
9
"testing"
10
+ "time"
11
)
12
13
func TestExample(t *testing.T) {
10
- out, err := exec.Command("go", "run", "main.go").Output()
14
+ t.Log("Starting go run main.go...")
15
+ start := time.Now()
16
+
17
+ cmd := exec.Command("go", "run", "main.go")
18
+ cmd.Env = append(os.Environ(), "GOLOG_LOG_LEVEL=error") // reduce libp2p noise
19
+
20
+ // Stream output to both test log and capture buffer for verification
21
+ // This ensures we see progress even if the process is killed
22
+ var buf bytes.Buffer
23
+ cmd.Stdout = io.MultiWriter(os.Stdout, &buf)
24
+ cmd.Stderr = io.MultiWriter(os.Stderr, &buf)
25
+
26
+ err := cmd.Run()
27
+
28
+ elapsed := time.Since(start)
29
+ t.Logf("Command completed in %v", elapsed)
30
+
31
+ out := buf.String()
32
if err != nil {
12
- var stderr string
13
- if xe, ok := err.(*exec.ExitError); ok {
14
- stderr = string(xe.Stderr)
15
- }
16
- t.Fatalf("running example (%v): %s\n%s", err, string(out), stderr)
33
+ t.Fatalf("running example (%v):\n%s", err, out)
34
}
18
- if !strings.Contains(string(out), "All done!") {
19
- t.Errorf("example did not run successfully")
35
+
36
+ if !strings.Contains(out, "All done!") {
37
+ t.Errorf("example did not complete successfully, output:\n%s", out)
38
}
39
}
mk/golang.mk
+41
-24
@@ -41,40 +41,57 @@ define go-build
41
$(GOCC) build $(go-flags-with-tags) -o "$@" "$(1)"
42
endef
43
44
-test_go_test: $$(DEPS_GO)
45
- $(GOCC) test $(go-flags-with-tags) $(GOTFLAGS) ./...
46
-.PHONY: test_go_test
47
-
48
-# Build all platforms from .github/build-platforms.yml
44
+# Only disable colors when running in CI (non-interactive terminal)
45
+GOTESTSUM_NOCOLOR := $(if $(CI),--no-color,)
46
+
47
+# Packages excluded from coverage (test code and examples are not production code)
48
+COVERPKG_EXCLUDE := /(test|docs/examples)/
49
+
50
+# Packages excluded from unit tests: coverage exclusions + client/rpc (tested by test_cli)
51
+UNIT_EXCLUDE := /(test|docs/examples)/|/client/rpc$$
52
+
53
+# Unit tests with coverage
54
+# Produces JSON for CI reporting and coverage profile for Codecov
55
+test_unit: test/bin/gotestsum $$(DEPS_GO)
56
+ mkdir -p test/unit coverage
57
+ rm -f test/unit/gotest.json coverage/unit_tests.coverprofile
58
+ gotestsum $(GOTESTSUM_NOCOLOR) --jsonfile test/unit/gotest.json -- $(go-flags-with-tags) $(GOTFLAGS) -covermode=atomic -coverprofile=coverage/unit_tests.coverprofile -coverpkg=$$($(GOCC) list $(go-tags) ./... | grep -vE '$(COVERPKG_EXCLUDE)' | tr '\n' ',' | sed 's/,$$//') $$($(GOCC) list $(go-tags) ./... | grep -vE '$(UNIT_EXCLUDE)')
59
+.PHONY: test_unit
60
+
61
+# CLI/integration tests (requires built binary in PATH)
62
+# Includes test/cli, test/integration, and client/rpc
63
+# Produces JSON for CI reporting
64
+# Override TEST_CLI_TIMEOUT for local development: make test_cli TEST_CLI_TIMEOUT=5m
65
+TEST_CLI_TIMEOUT ?= 10m
66
+test_cli: cmd/ipfs/ipfs test/bin/gotestsum $$(DEPS_GO)
67
+ mkdir -p test/cli
68
+ rm -f test/cli/cli-tests.json
69
+ PATH="$(CURDIR)/cmd/ipfs:$(CURDIR)/test/bin:$$PATH" gotestsum $(GOTESTSUM_NOCOLOR) --jsonfile test/cli/cli-tests.json -- -v -timeout=$(TEST_CLI_TIMEOUT) ./test/cli/... ./test/integration/... ./client/rpc/...
70
+.PHONY: test_cli
71
+
72
+# Example tests (docs/examples/kubo-as-a-library)
73
+# Tests against both published and current kubo versions
74
+# Uses timeout to ensure CI gets output before job-level timeout kills everything
75
+TEST_EXAMPLES_TIMEOUT ?= 2m
76
+test_examples:
77
+ cd docs/examples/kubo-as-a-library && go test -v -timeout=$(TEST_EXAMPLES_TIMEOUT) ./... && cp go.mod go.mod.bak && cp go.sum go.sum.bak && (go mod edit -replace github.com/ipfs/kubo=./../../.. && go mod tidy && go test -v -timeout=$(TEST_EXAMPLES_TIMEOUT) ./...; ret=$$?; mv go.mod.bak go.mod; mv go.sum.bak go.sum; exit $$ret)
78
+.PHONY: test_examples
79
+
80
+# Build kubo for all platforms from .github/build-platforms.yml
81
test_go_build:
82
bin/test-go-build-platforms
83
.PHONY: test_go_build
84
53
-test_go_short: GOTFLAGS += -test.short
54
-test_go_short: test_go_test
55
-.PHONY: test_go_short
56
-
57
-test_go_race: GOTFLAGS += -race
58
-test_go_race: test_go_test
59
-.PHONY: test_go_race
60
-
61
-test_go_expensive: test_go_test test_go_build
62
-.PHONY: test_go_expensive
63
-TEST_GO += test_go_expensive
64
-
85
+# Check Go source formatting
86
test_go_fmt:
87
bin/test-go-fmt
88
.PHONY: test_go_fmt
68
-TEST_GO += test_go_fmt
89
90
+# Run golangci-lint (used by CI)
91
test_go_lint: test/bin/golangci-lint
92
golangci-lint run --timeout=3m ./...
93
.PHONY: test_go_lint
94
74
-test_go: $(TEST_GO)
75
-
76
-# Version check is no longer needed - go.mod enforces minimum version
77
-.PHONY: check_go_version
78
-
95
+TEST_GO := test_go_fmt test_unit test_cli test_examples
96
TEST += $(TEST_GO)
80
-TEST_SHORT += test_go_fmt test_go_short
97
+TEST_SHORT += test_go_fmt test_unit
test/cli/delegated_routing_v1_http_server_test.go
+23
-40
@@ -2,7 +2,6 @@ package cli
2
3
import (
4
"context"
5
- "encoding/json"
5
"strings"
6
"testing"
7
"time"
@@ -21,11 +20,6 @@ import (
20
"github.com/stretchr/testify/require"
21
)
22
24
-// swarmPeersOutput is used to parse the JSON output of 'ipfs swarm peers --enc=json'
25
-type swarmPeersOutput struct {
26
- Peers []struct{} `json:"Peers"`
27
-}
28
-
23
func TestRoutingV1Server(t *testing.T) {
24
t.Parallel()
25
@@ -206,11 +200,14 @@ func TestRoutingV1Server(t *testing.T) {
200
c, err := client.New(node.GatewayURL())
201
require.NoError(t, err)
202
209
- // Try to get closest peers - should fail gracefully with an error
203
+ // Try to get closest peers - should fail gracefully with an error.
204
+ // Use 60-second timeout (server has 30s routing timeout).
205
testCid, err := cid.Decode("QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn")
206
require.NoError(t, err)
207
213
- _, err = c.GetClosestPeers(context.Background(), testCid)
208
+ ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
209
+ defer cancel()
210
+ _, err = c.GetClosestPeers(ctx, testCid)
211
require.Error(t, err)
212
// All these routing types should indicate DHT is not available
213
// The exact error message may vary based on implementation details
@@ -224,7 +221,7 @@ func TestRoutingV1Server(t *testing.T) {
221
}
222
})
223
227
- t.Run("GetClosestPeers returns peers for self", func(t *testing.T) {
224
+ t.Run("GetClosestPeers returns peers", func(t *testing.T) {
225
t.Parallel()
226
227
routingTypes := []string{"auto", "autoclient", "dht", "dhtclient"}
@@ -242,47 +239,33 @@ func TestRoutingV1Server(t *testing.T) {
239
})
240
node.StartDaemon()
241
245
- // Create client before waiting so we can probe DHT readiness
242
c, err := client.New(node.GatewayURL())
243
require.NoError(t, err)
244
245
// Query for closest peers to our own peer ID
246
key := peer.ToCid(node.PeerID())
247
252
- // Wait for node to connect to bootstrap peers and populate WAN DHT routing table
253
- minPeers := len(autoconf.FallbackBootstrapPeers)
254
- require.EventuallyWithT(t, func(t *assert.CollectT) {
255
- res := node.RunIPFS("swarm", "peers", "--enc=json")
256
- var output swarmPeersOutput
257
- err := json.Unmarshal(res.Stdout.Bytes(), &output)
258
- assert.NoError(t, err)
259
- peerCount := len(output.Peers)
260
- // Wait until we have at least minPeers connected
261
- assert.GreaterOrEqual(t, peerCount, minPeers,
262
- "waiting for at least %d bootstrap peers, currently have %d", minPeers, peerCount)
263
- }, 60*time.Second, time.Second)
264
-
265
- // Wait for DHT to be ready by probing GetClosestPeers until it succeeds
266
- require.EventuallyWithT(t, func(t *assert.CollectT) {
267
- probeCtx, probeCancel := context.WithTimeout(context.Background(), 30*time.Second)
268
- defer probeCancel()
269
- probeIter, probeErr := c.GetClosestPeers(probeCtx, key)
270
- if probeErr == nil {
271
- probeIter.Close()
248
+ // Wait for WAN DHT routing table to be populated.
249
+ // The server has a 30-second routing timeout, so we use 60 seconds
250
+ // per request to allow for network latency while preventing hangs.
251
+ // Total wait time is 2 minutes (locally passes in under 1 minute).
252
+ var records []*types.PeerRecord
253
+ require.EventuallyWithT(t, func(ct *assert.CollectT) {
254
+ ctx, cancel := context.WithTimeout(t.Context(), 60*time.Second)
255
+ defer cancel()
256
+ resultsIter, err := c.GetClosestPeers(ctx, key)
257
+ if !assert.NoError(ct, err) {
258
+ return
259
}
273
- assert.NoError(t, probeErr, "DHT should be ready to handle GetClosestPeers")
260
+ records, err = iter.ReadAllResults(resultsIter)
261
+ assert.NoError(ct, err)
262
}, 2*time.Minute, 5*time.Second)
263
276
- ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
277
- defer cancel()
278
- resultsIter, err := c.GetClosestPeers(ctx, key)
279
- require.NoError(t, err)
280
-
281
- records, err := iter.ReadAllResults(resultsIter)
282
- require.NoError(t, err)
283
-
264
// Verify we got some peers back from WAN DHT
285
- assert.NotEmpty(t, records, "should return some peers close to own peerid")
265
+ require.NotEmpty(t, records, "should return peers close to own peerid")
266
+
267
+ // Per IPIP-0476, GetClosestPeers returns at most 20 peers
268
+ assert.LessOrEqual(t, len(records), 20, "IPIP-0476 limits GetClosestPeers to 20 peers")
269
270
// Verify structure of returned records
271
for _, record := range records {
test/dependencies/go.mod
+1
-1
@@ -15,7 +15,7 @@ require (
15
github.com/ipfs/iptb-plugins v0.5.1
16
github.com/multiformats/go-multiaddr v0.16.1
17
github.com/multiformats/go-multihash v0.2.3
18
- gotest.tools/gotestsum v1.12.3
18
+ gotest.tools/gotestsum v1.13.0
19
)
20
21
require (
test/dependencies/go.sum
+2
-2
@@ -985,8 +985,8 @@ gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
985
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
986
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
987
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
988
-gotest.tools/gotestsum v1.12.3 h1:jFwenGJ0RnPkuKh2VzAYl1mDOJgbhobBDeL2W1iEycs=
989
-gotest.tools/gotestsum v1.12.3/go.mod h1:Y1+e0Iig4xIRtdmYbEV7K7H6spnjc1fX4BOuUhWw2Wk=
988
+gotest.tools/gotestsum v1.13.0 h1:+Lh454O9mu9AMG1APV4o0y7oDYKyik/3kBOiCqiEpRo=
989
+gotest.tools/gotestsum v1.13.0/go.mod h1:7f0NS5hFb0dWr4NtcsAsF0y1kzjEFfAil0HiBQJE03Q=
990
gotest.tools/v3 v3.5.2 h1:7koQfIKdy+I8UTetycgUqXWSDwpgv193Ka+qRsmBY8Q=
991
gotest.tools/v3 v3.5.2/go.mod h1:LtdLGcnqToBH83WByAAi/wiwSFCArdFIUV/xxN4pcjA=
992
honnef.co/go/tools v0.6.1 h1:R094WgE8K4JirYjBaOpz/AvTyUu/3wbmAoskKN/pxTI=
test/unit/Rules.mk
+2
-1
@@ -2,7 +2,8 @@ include mk/header.mk
2
3
CLEAN += $(d)/gotest.json $(d)/gotest.junit.xml
4
5
-$(d)/gotest.junit.xml: test/bin/gotestsum coverage/unit_tests.coverprofile
5
+# Convert gotest.json (produced by test_unit) to JUnit XML format
6
+$(d)/gotest.junit.xml: test/bin/gotestsum $(d)/gotest.json
7
gotestsum --no-color --junitfile $@ --raw-command cat $(@D)/gotest.json
8
9
include mk/footer.mk