@cryptotaxi247 / kubo / commits / 614b75753

feat: bound graceful shutdown, add diag healthy (#11329)

* feat: bound graceful shutdown, add diag healthy Replace unbounded app.Stop(context.Background()) with a deadline-bounded context driven by a new Internal.ShutdownTimeout config (default 12h, 0 disables). Add an os.Exit(1) watchdog at the same deadline so an FX OnStop hook that never returns can no longer hang the daemon. Add ipfs diag healthy: fails when shutdown has been initiated or when the DAG pipeline cannot resolve the well-known empty-directory CID. Dockerfile HEALTHCHECK now uses it so orchestrators recycle half- shutdown daemons. - core/shutdown: new pkg; atomic startedAt + CloseWithCtx helper - core/builder.go: app.Stop bounded by ShutdownTimeout - cmd/ipfs/kubo/daemon.go: watchdog + MarkStarted on signal - core/commands/diag.go: new healthy subcommand - core/node/{bitswap,libp2p/host,libp2p/routing}.go: OnStop hooks wrapped - config/internal.go: ShutdownTimeout + DefaultShutdownTimeout=12h - Dockerfile: HEALTHCHECK uses "ipfs diag healthy" - docs/{config,changelogs/v0.42}.md: documented - test/cli: enabled + disabled path tests * feat: bound provider stats and ADD_PROVIDER sends bumps go-libp2p-kad-dht past v0.39.2 to b73e1e8 to pick up two related provider bug fixes. - ipfs provide stat now honors client cancellation and deadlines instead of blocking indefinitely behind a slow keystore lookup - adds Provide.DHT.SendProviderRecordTimeout capping each ADD_PROVIDER RPC so unresponsive peers cannot pin a provide worker and stall reprovide cycles - internal reprovide-alert poller bounds its Stats call so a hung keystore.Size cannot delay shutdown * test(shutdown): use synctest for timeout test, document sleep CloseWithCtx_timesOut now runs in a synctest bubble so the deadline assertion is exact (no wall-clock slack), and the simulated close uses a release channel to drain the bubble cleanly after the leak point. The two happy-path tests stay unchanged because their close funcs return immediately and gain nothing from a fake clock. Comment the 2ms sleep in TestMarkStartedPreservesFirstTimestamp so its role (forcing time.Now() to advance between the two MarkStarted calls so a CAS to Store regression is detectable) is not lost. Addresses ipfs/kubo#11329 (review). * fix(pinner): bound pinner Close with shutdown deadline The boxo Pinner.Close contract notes that an in-flight op ignoring its ctx (a downstream bug) can block Close, so the host must bound it at the call site. Wrapping the OnStop hook with CloseWithCtx honors Internal.ShutdownTimeout and surfaces an actionable "subsystem 'pinner' failed to close" log on hang instead of leaving only the watchdog os.Exit(1) trace. * fix(shutdown): bound remaining I/O-touching OnStop hooks Wrap the OnStop hooks whose Close can plausibly block on disk or network: repo (datastore flush + lock release), mfs-root (datastore writes via DAGService), peering (waits on libp2p peer goroutines), legacy-provider (in-flight reprovide RPCs), and the dht-provider plus keystore pair under SweepingProvider. In-memory closes (blockservice, peerstore, resource-manager) are left as-is since they cannot realistically hang. For the dht-provider/keystore pair, provider closes first so nothing can access the keystore afterwards. If the shutdown ctx fires mid-provider-drain, the keystore close sees an expired ctx and returns immediately; the watchdog os.Exit(1) is the ultimate backstop, and keystore writes are fsync'd on put so missing the explicit close is recoverable on next boot. * fix(shutdown): bound remaining in-memory OnStop hooks Wrap blockservice, peerstore, and resource-manager Close hooks with CloseWithCtx for uniformity. These are pure in-memory operations unlikely to hang in practice, but wrapping costs nothing and makes the shutdown audit trail uniform: every OnStop hook now honors the deadline and surfaces a named subsystem on timeout. * fix(shutdown): bound autoRelayFeeder OnStop on ctx OnStop waited on the feeder goroutine via <-done without honoring the shutdown ctx. The goroutine itself selects on ctx in every loop case, so cancel() normally suffices, but a stuck downstream dht.WAN.GetClosestPeers that ignored its ctx could block fx.Stop indefinitely. Adding the ctx.Done() select case mirrors the reprovideAlert pattern in provider.go and lets the shutdown deadline reclaim control even with a misbehaving DHT. * docs(changelog): merge shutdown entries into one user-facing section Combine the pinner-on-shutdown paragraph with the bounded-shutdown section under a single "Reliable shutdown and container health checks" heading. Lead with the visible symptoms (half-shutdown daemons, healthy-but-dead container reports, manual docker restart) instead of fx OnStop jargon. Frame Internal.ShutdownTimeout as a belt-and-suspenders ceiling, with the 12-hour default sized against the 22-hour DHT provider record expiration.

Marcin Rataj committed May 15, 2026 at 00:54 UTC 614b75753221ad6db73ed0b1a51caa6150298f1f
30 files changed +549 -54
Dockerfile
+3 -3
@@ -97,10 +97,10 @@ ENV GOLOG_LOG_LEVEL=""
97 # tini ensures proper signal handling and zombie process cleanup
98 ENTRYPOINT ["/sbin/tini", "--", "/usr/local/bin/start_ipfs"]
99
100 -# Health check verifies IPFS daemon is responsive.
101 -# Uses empty directory CID (QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn) as test
100 +# Health check via "ipfs diag healthy": verifies RPC + DAG pipeline, and
101 +# fails after SIGINT/SIGTERM to catch half-shutdown states.
102 HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
103 - CMD ipfs --api=/ip4/127.0.0.1/tcp/5001 dag stat /ipfs/QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn || exit 1
103 + CMD ipfs --api=/ip4/127.0.0.1/tcp/5001 diag healthy > /dev/null 2>&1 || exit 1
104
105 # Default: run IPFS daemon with auto-migration enabled
106 CMD ["daemon", "--migrate=true", "--agent-version-suffix=docker"]
cmd/ipfs/kubo/daemon.go
+20
@@ -31,6 +31,7 @@ import (
31 options "github.com/ipfs/kubo/core/coreiface/options"
32 corerepo "github.com/ipfs/kubo/core/corerepo"
33 libp2p "github.com/ipfs/kubo/core/node/libp2p"
34 + "github.com/ipfs/kubo/core/shutdown"
35 nodeMount "github.com/ipfs/kubo/fuse/node"
36 fsrepo "github.com/ipfs/kubo/repo/fsrepo"
37 "github.com/ipfs/kubo/repo/fsrepo/migrations"
@@ -432,6 +433,12 @@ func daemonFunc(req *cmds.Request, re cmds.ResponseEmitter, env cmds.Environment
433 ipnsps = cfg.Ipns.UsePubsub.WithDefault(false)
434 }
435
436 + // Resolve graceful-shutdown timeout. The generous 12h default leaves
437 + // normal operation unchanged while guaranteeing the daemon cannot be
438 + // stuck indefinitely on a hung FX OnStop hook. A value of 0 opts out
439 + // entirely and restores the legacy "wait forever" behavior.
440 + shutdownTimeout := max(cfg.Internal.ShutdownTimeout.WithDefault(config.DefaultShutdownTimeout), 0)
441 +
442 // Start assembling node config
443 ncfg := &core.BuildCfg{
444 Repo: repo,
@@ -442,6 +449,7 @@ func daemonFunc(req *cmds.Request, re cmds.ResponseEmitter, env cmds.Environment
449 "pubsub": pubsub,
450 "ipnsps": ipnsps,
451 },
452 + ShutdownTimeout: shutdownTimeout,
453 // TODO(Kubuxu): refactor Online vs Offline by adding Permanent vs Ephemeral
454 }
455
@@ -583,6 +591,17 @@ take effect.
591 }
592
593 defer func() {
594 + // Watchdog: if node.Close() does not return within shutdownTimeout,
595 + // force-exit so orchestrators can restart the daemon.
596 + // shutdownTimeout==0 disables the watchdog (wait forever).
597 + if shutdownTimeout > 0 {
598 + killSwitch := time.AfterFunc(shutdownTimeout, func() {
599 + log.Errorf("shutdown watchdog: node.Close() did not return after %s; exiting", shutdownTimeout)
600 + os.Exit(1)
601 + })
602 + defer killSwitch.Stop()
603 + }
604 +
605 // We wait for the node to close first, as the node has children
606 // that it will wait for before closing, such as the API server.
607 node.Close()
@@ -708,6 +727,7 @@ take effect.
727 // Give the user some immediate feedback when they hit C-c
728 go func() {
729 <-req.Context.Done()
730 + shutdown.MarkStarted()
731 notifyStopping()
732 fmt.Println("Received interrupt signal, shutting down...")
733 fmt.Println("(Hit ctrl-c again to force-shutdown the daemon.)")
config/internal.go
+16
@@ -1,8 +1,19 @@
1 package config
2
3 +import "time"
4 +
5 const (
6 // DefaultMFSNoFlushLimit is the default limit for consecutive unflushed MFS operations
7 DefaultMFSNoFlushLimit = 256
8 +
9 + // DefaultShutdownTimeout caps how long graceful shutdown is allowed to
10 + // take before the daemon force-exits with status 1. Set generously so
11 + // it does not change existing kubo behavior in practice but guarantees
12 + // Docker / kubernetes infrastructure can never be stuck indefinitely
13 + // on a hung FX OnStop hook. Smaller than the 22h DHT reprovide cycle,
14 + // so a hung daemon recovers before missing more than one cycle.
15 + // Set Internal.ShutdownTimeout to 0 to opt out and wait forever.
16 + DefaultShutdownTimeout = 12 * time.Hour
17 )
18
19 type Internal struct {
@@ -18,6 +29,11 @@ type Internal struct {
29 // This is an EXPERIMENTAL feature and may change or be removed in future releases.
30 // See https://github.com/ipfs/kubo/issues/10842
31 MFSNoFlushLimit *OptionalInteger `json:",omitempty"`
32 + // ShutdownTimeout caps how long graceful shutdown of the daemon is
33 + // allowed to take. Defaults to DefaultShutdownTimeout. When the
34 + // deadline expires the daemon logs which subsystem failed to close and
35 + // exits with status 1. Set to 0 to disable the cap and wait forever.
36 + ShutdownTimeout *OptionalDuration `json:",omitempty"`
37 }
38
39 type InternalBitswap struct {
config/provide.go
+25 -9
@@ -29,15 +29,16 @@ const (
29 MinProvideBloomFPRate = 1_000_000
30
31 // DHT provider defaults
32 - DefaultProvideDHTInterval = 22 * time.Hour // https://github.com/ipfs/kubo/pull/9326
33 - DefaultProvideDHTMaxWorkers = 16 // Unified default for both sweep and legacy providers
34 - DefaultProvideDHTSweepEnabled = true
35 - DefaultProvideDHTResumeEnabled = true
36 - DefaultProvideDHTDedicatedPeriodicWorkers = 2
37 - DefaultProvideDHTDedicatedBurstWorkers = 1
38 - DefaultProvideDHTMaxProvideConnsPerWorker = 20
39 - DefaultProvideDHTKeystoreBatchSize = 1 << 14 // ~544 KiB per batch (1 multihash = 34 bytes)
40 - DefaultProvideDHTOfflineDelay = 2 * time.Hour
32 + DefaultProvideDHTInterval = 22 * time.Hour // https://github.com/ipfs/kubo/pull/9326
33 + DefaultProvideDHTMaxWorkers = 16 // Unified default for both sweep and legacy providers
34 + DefaultProvideDHTSweepEnabled = true
35 + DefaultProvideDHTResumeEnabled = true
36 + DefaultProvideDHTDedicatedPeriodicWorkers = 2
37 + DefaultProvideDHTDedicatedBurstWorkers = 1
38 + DefaultProvideDHTMaxProvideConnsPerWorker = 20
39 + DefaultProvideDHTKeystoreBatchSize = 1 << 14 // ~544 KiB per batch (1 multihash = 34 bytes)
40 + DefaultProvideDHTOfflineDelay = 2 * time.Hour
41 + DefaultProvideDHTSendProviderRecordTimeout = 10 * time.Second
42
43 // DefaultFastProvideTimeout is the maximum time allowed for fast-provide operations.
44 // Prevents hanging on network issues when providing root CID.
@@ -121,6 +122,13 @@ type ProvideDHT struct {
122 // Default: DefaultProvideDHTOfflineDelay
123 OfflineDelay *OptionalDuration `json:",omitempty"`
124
125 + // SendProviderRecordTimeout sets the per-peer timeout applied to a single
126 + // ADD_PROVIDER RPC. A peer that accepts the libp2p stream but never reads
127 + // the request must not pin a provide worker goroutine indefinitely; this
128 + // timeout bounds the wait (sweep mode only).
129 + // Default: DefaultProvideDHTSendProviderRecordTimeout
130 + SendProviderRecordTimeout *OptionalDuration `json:",omitempty"`
131 +
132 // ResumeEnabled controls whether the provider resumes from its previous state on restart.
133 // When enabled, the provider persists its reprovide cycle state and provide queue to the datastore,
134 // and restores them on restart. When disabled, the provider starts fresh on each restart.
@@ -259,6 +267,14 @@ func ValidateProvideConfig(cfg *Provide) error {
267 }
268 }
269
270 + // Validate SendProviderRecordTimeout
271 + if !cfg.DHT.SendProviderRecordTimeout.IsDefault() {
272 + timeout := cfg.DHT.SendProviderRecordTimeout.WithDefault(DefaultProvideDHTSendProviderRecordTimeout)
273 + if timeout <= 0 {
274 + return fmt.Errorf("Provide.DHT.SendProviderRecordTimeout must be positive, got %v", timeout)
275 + }
276 + }
277 +
278 return nil
279 }
280
core/builder.go
+14 -2
@@ -93,9 +93,21 @@ func NewNode(ctx context.Context, cfg *BuildCfg) (*IpfsNode, error) {
93 var stopErr error
94 n.stop = func() error {
95 once.Do(func() {
96 - stopErr = app.Stop(context.Background())
96 + // Bound app.Stop with a deadline so an FX OnStop hook that
97 + // never returns cannot hang the daemon. ShutdownTimeout==0
98 + // opts out of the cap entirely and restores the legacy
99 + // behavior of waiting forever for hooks to complete. The
100 + // daemon's watchdog in cmd/ipfs/kubo/daemon.go fires at the
101 + // same deadline and is the unconditional os.Exit fallback.
102 + stopCtx := context.Background()
103 + if cfg.ShutdownTimeout > 0 {
104 + var stopCancel context.CancelFunc
105 + stopCtx, stopCancel = context.WithTimeout(stopCtx, cfg.ShutdownTimeout)
106 + defer stopCancel()
107 + }
108 + stopErr = app.Stop(stopCtx)
109 if stopErr != nil {
98 - log.Error("failure on stop: ", stopErr)
110 + log.Errorf("failure on stop: %v", stopErr)
111 }
112 // Cancel the context _after_ the app has stopped.
113 cancel()
core/commands/commands_test.go
+1
@@ -81,6 +81,7 @@ func TestCommands(t *testing.T) {
81 "/diag/datastore/count",
82 "/diag/datastore/get",
83 "/diag/datastore/put",
84 + "/diag/healthy",
85 "/diag/profile",
86 "/diag/sys",
87 "/files",
core/commands/diag.go
+45
@@ -5,16 +5,26 @@ import (
5 "errors"
6 "fmt"
7 "io"
8 + "time"
9
10 + "github.com/ipfs/boxo/path"
11 + cid "github.com/ipfs/go-cid"
12 "github.com/ipfs/go-datastore"
13 "github.com/ipfs/go-datastore/mount"
14 "github.com/ipfs/go-datastore/query"
15 cmds "github.com/ipfs/go-ipfs-cmds"
16 oldcmds "github.com/ipfs/kubo/commands"
17 + "github.com/ipfs/kubo/core/commands/cmdenv"
18 node "github.com/ipfs/kubo/core/node"
19 + "github.com/ipfs/kubo/core/shutdown"
20 fsrepo "github.com/ipfs/kubo/repo/fsrepo"
21 )
22
23 +// diagHealthyProbeCIDStr is the well-known empty UnixFS directory,
24 +// built into every kubo node. Fetching it succeeds regardless of peers,
25 +// DHT, or user content, so it isolates the DAG/blockstore pipeline.
26 +const diagHealthyProbeCIDStr = "QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn"
27 +
28 var DiagCmd = &cmds.Command{
29 Helptext: cmds.HelpText{
30 Tagline: "Generate diagnostic reports.",
@@ -25,6 +35,41 @@ var DiagCmd = &cmds.Command{
35 "cmds": ActiveReqsCmd,
36 "profile": sysProfileCmd,
37 "datastore": diagDatastoreCmd,
38 + "healthy": diagHealthyCmd,
39 + },
40 +}
41 +
42 +// diagHealthyCmd is a container-healthcheck probe. It fails when shutdown
43 +// has been initiated (even if the RPC API still answers) or when the DAG
44 +// pipeline cannot resolve a built-in CID.
45 +var diagHealthyCmd = &cmds.Command{
46 + Helptext: cmds.HelpText{
47 + Tagline: "Report whether the daemon is operational.",
48 + ShortDescription: `
49 +Exits 0 if the daemon is running and can resolve the well-known empty
50 +UnixFS directory. Exits non-zero if shutdown has started or the DAG
51 +pipeline is broken. Intended for container healthchecks.
52 +`,
53 + },
54 + Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) error {
55 + if t := shutdown.StartedAt(); !t.IsZero() {
56 + return fmt.Errorf("daemon is shutting down (started %s ago)", time.Since(t).Round(time.Second))
57 + }
58 + api, err := cmdenv.GetApi(env, req)
59 + if err != nil {
60 + return err
61 + }
62 + probeCID, err := cid.Decode(diagHealthyProbeCIDStr)
63 + if err != nil {
64 + return fmt.Errorf("invalid probe CID: %w", err)
65 + }
66 + if _, _, err := api.ResolvePath(req.Context, path.FromCid(probeCID)); err != nil {
67 + return fmt.Errorf("probe resolve: %w", err)
68 + }
69 + if _, err := api.Dag().Get(req.Context, probeCID); err != nil {
70 + return fmt.Errorf("probe fetch: %w", err)
71 + }
72 + return cmds.EmitOnce(res, "ok")
73 },
74 }
75
core/commands/provide.go
+4 -1
@@ -267,7 +267,10 @@ NOTES:
267 return fmt.Errorf("stats not available with current routing system %T", nd.Provider)
268 }
269
270 - s := sweepingProvider.Stats()
270 + s, err := sweepingProvider.Stats(req.Context)
271 + if err != nil {
272 + return err
273 + }
274 return res.Emit(provideStats{Sweep: &s})
275 },
276 Encoders: cmds.EncoderMap{
core/node/bitswap.go
+3 -2
@@ -26,6 +26,7 @@ import (
26
27 blocks "github.com/ipfs/go-block-format"
28 "github.com/ipfs/kubo/core/node/helpers"
29 + "github.com/ipfs/kubo/core/shutdown"
30 )
31
32 // Docs: https://github.com/ipfs/kubo/blob/master/docs/config.md#internalbitswap
@@ -197,7 +198,7 @@ func Bitswap(serverEnabled, libp2pEnabled, httpEnabled bool) any {
198
199 lc.Append(fx.Hook{
200 OnStop: func(ctx context.Context) error {
200 - return bs.Close()
201 + return shutdown.CloseWithCtx(ctx, "bitswap", bs.Close)
202 },
203 })
204 return bs, nil
@@ -213,7 +214,7 @@ func OnlineExchange(isBitswapActive bool) any {
214 }
215 lc.Append(fx.Hook{
216 OnStop: func(ctx context.Context) error {
216 - return in.Close()
217 + return shutdown.CloseWithCtx(ctx, "bitswap-exchange", in.Close)
218 },
219 })
220 return in
core/node/builder.go
+8 -1
@@ -4,12 +4,14 @@ import (
4 "context"
5 "crypto/rand"
6 "encoding/base64"
7 + "time"
8
9 "go.uber.org/fx"
10
11 "github.com/ipfs/boxo/autoconf"
12 "github.com/ipfs/kubo/core/node/helpers"
13 "github.com/ipfs/kubo/core/node/libp2p"
14 + "github.com/ipfs/kubo/core/shutdown"
15 "github.com/ipfs/kubo/repo"
16
17 ds "github.com/ipfs/go-datastore"
@@ -37,6 +39,11 @@ type BuildCfg struct {
39 Routing libp2p.RoutingOption
40 Host libp2p.HostOption
41 Repo repo.Repo
42 +
43 + // ShutdownTimeout caps how long node.Close()'s call to app.Stop is
44 + // allowed to take. Zero disables the cap (app.Stop runs with no
45 + // deadline, matching the legacy "wait forever" behavior).
46 + ShutdownTimeout time.Duration
47 }
48
49 func (cfg *BuildCfg) getOpt(key string) bool {
@@ -77,7 +84,7 @@ func (cfg *BuildCfg) options(ctx context.Context) (fx.Option, *cfg.Config) {
84 repoOption := fx.Provide(func(lc fx.Lifecycle) repo.Repo {
85 lc.Append(fx.Hook{
86 OnStop: func(ctx context.Context) error {
80 - return cfg.Repo.Close()
87 + return shutdown.CloseWithCtx(ctx, "repo", cfg.Repo.Close)
88 },
89 })
90
core/node/core.go
+10 -4
@@ -27,6 +27,7 @@ import (
27
28 "github.com/ipfs/kubo/config"
29 "github.com/ipfs/kubo/core/node/helpers"
30 + "github.com/ipfs/kubo/core/shutdown"
31 "github.com/ipfs/kubo/repo"
32 )
33
@@ -42,7 +43,7 @@ func BlockService(cfg *config.Config) func(lc fx.Lifecycle, bs blockstore.Blocks
43
44 lc.Append(fx.Hook{
45 OnStop: func(ctx context.Context) error {
45 - return bsvc.Close()
46 + return shutdown.CloseWithCtx(ctx, "blockservice", bsvc.Close)
47 },
48 })
49
@@ -103,9 +104,14 @@ func Pinning(strategy string) func(lc fx.Lifecycle, bstore blockstore.Blockstore
104 // repo provider registers its close hook earlier (in
105 // builder.go), so this hook runs first and the repo hook
106 // runs after, without an explicit dependency between them.
107 + //
108 + // Wrapped with CloseWithCtx because the boxo Pinner.Close
109 + // contract notes that an in-flight op which ignores its ctx
110 + // (a downstream bug) can block Close; the host must bound it
111 + // at the call site so the shutdown deadline is honored.
112 lc.Append(fx.Hook{
107 - OnStop: func(context.Context) error {
108 - return pinning.Close()
113 + OnStop: func(ctx context.Context) error {
114 + return shutdown.CloseWithCtx(ctx, "pinner", pinning.Close)
115 },
116 })
117
@@ -276,7 +282,7 @@ func Files(strategy string) func(mctx helpers.MetricsCtx, lc fx.Lifecycle, repo
282
283 lc.Append(fx.Hook{
284 OnStop: func(ctx context.Context) error {
279 - return root.Close()
285 + return shutdown.CloseWithCtx(ctx, "mfs-root", root.Close)
286 },
287 })
288
core/node/libp2p/host.go
+5 -1
@@ -13,6 +13,7 @@ import (
13
14 "github.com/ipfs/kubo/config"
15 "github.com/ipfs/kubo/core/node/helpers"
16 + "github.com/ipfs/kubo/core/shutdown"
17 "github.com/ipfs/kubo/repo"
18
19 "go.uber.org/fx"
@@ -104,7 +105,10 @@ func Host(mctx helpers.MetricsCtx, lc fx.Lifecycle, params P2PHostIn) (out P2PHo
105
106 lc.Append(fx.Hook{
107 OnStop: func(ctx context.Context) error {
107 - return out.Host.Close()
108 + // Host.Close() does not accept a ctx and can block draining
109 + // peer connections on busy nodes. CloseWithCtx returns when
110 + // either the close finishes or the shutdown deadline expires.
111 + return shutdown.CloseWithCtx(ctx, "libp2p-host", out.Host.Close)
112 },
113 })
114
core/node/libp2p/peerstore.go
+2 -1
@@ -3,6 +3,7 @@ package libp2p
3 import (
4 "context"
5
6 + "github.com/ipfs/kubo/core/shutdown"
7 "github.com/libp2p/go-libp2p/core/peerstore"
8 "github.com/libp2p/go-libp2p/p2p/host/peerstore/pstoremem"
9 "go.uber.org/fx"
@@ -15,7 +16,7 @@ func Peerstore(lc fx.Lifecycle) (peerstore.Peerstore, error) {
16 }
17 lc.Append(fx.Hook{
18 OnStop: func(ctx context.Context) error {
18 - return pstore.Close()
19 + return shutdown.CloseWithCtx(ctx, "peerstore", pstore.Close)
20 },
21 })
22
core/node/libp2p/rcmgr.go
+3 -2
@@ -10,6 +10,7 @@ import (
10
11 "github.com/ipfs/kubo/config"
12 "github.com/ipfs/kubo/core/node/helpers"
13 + "github.com/ipfs/kubo/core/shutdown"
14 "github.com/ipfs/kubo/repo"
15
16 logging "github.com/ipfs/go-log/v2"
@@ -124,8 +125,8 @@ filled in with autocomputed defaults.`)
125 opts.Opts = append(opts.Opts, libp2p.ResourceManager(manager))
126
127 lc.Append(fx.Hook{
127 - OnStop: func(_ context.Context) error {
128 - return manager.Close()
128 + OnStop: func(ctx context.Context) error {
129 + return shutdown.CloseWithCtx(ctx, "resource-manager", manager.Close)
130 },
131 })
132
core/node/libp2p/routing.go
+15 -6
@@ -24,6 +24,7 @@ import (
24
25 config "github.com/ipfs/kubo/config"
26 "github.com/ipfs/kubo/core/node/helpers"
27 + "github.com/ipfs/kubo/core/shutdown"
28 "github.com/ipfs/kubo/repo"
29 irouting "github.com/ipfs/kubo/routing"
30 )
@@ -71,7 +72,7 @@ func BaseRouting(cfg *config.Config) any {
72
73 lc.Append(fx.Hook{
74 OnStop: func(ctx context.Context) error {
74 - return dualDHT.Close()
75 + return shutdown.CloseWithCtx(ctx, "dht-dual", dualDHT.Close)
76 },
77 })
78 }
@@ -82,7 +83,7 @@ func BaseRouting(cfg *config.Config) any {
83 dualDHT = dht
84 lc.Append(fx.Hook{
85 OnStop: func(ctx context.Context) error {
85 - return dualDHT.Close()
86 + return shutdown.CloseWithCtx(ctx, "dht-dual-composable", dualDHT.Close)
87 },
88 })
89 break
@@ -116,7 +117,7 @@ func BaseRouting(cfg *config.Config) any {
117
118 lc.Append(fx.Hook{
119 OnStop: func(ctx context.Context) error {
119 - return fullRTClient.Close()
120 + return shutdown.CloseWithCtx(ctx, "dht-fullrt", fullRTClient.Close)
121 },
122 })
123
@@ -337,10 +338,18 @@ func autoRelayFeeder(cfgPeering config.Peering, peerChan chan<- peer.AddrInfo) f
338 }()
339
340 lc.Append(fx.Hook{
340 - OnStop: func(_ context.Context) error {
341 + OnStop: func(ctx context.Context) error {
342 cancel()
342 - <-done
343 - return nil
343 + // Wait for the feeder goroutine to exit but bound by
344 + // the shutdown deadline so a stuck DHT call (downstream
345 + // bug ignoring ctx) cannot block fx.Stop. Mirrors the
346 + // reprovideAlert pattern in provider.go.
347 + select {
348 + case <-done:
349 + return nil
350 + case <-ctx.Done():
351 + return ctx.Err()
352 + }
353 },
354 })
355 })
core/node/peering.go
+6 -3
@@ -4,6 +4,7 @@ import (
4 "context"
5
6 "github.com/ipfs/boxo/peering"
7 + "github.com/ipfs/kubo/core/shutdown"
8 "github.com/libp2p/go-libp2p/core/host"
9 "github.com/libp2p/go-libp2p/core/peer"
10 "go.uber.org/fx"
@@ -17,9 +18,11 @@ func Peering(lc fx.Lifecycle, host host.Host) *peering.PeeringService {
18 OnStart: func(context.Context) error {
19 return ps.Start()
20 },
20 - OnStop: func(context.Context) error {
21 - ps.Stop()
22 - return nil
21 + OnStop: func(ctx context.Context) error {
22 + return shutdown.CloseWithCtx(ctx, "peering", func() error {
23 + ps.Stop()
24 + return nil
25 + })
26 },
27 })
28 return ps
core/node/provider.go
+21 -8
@@ -23,6 +23,7 @@ import (
23 "github.com/ipfs/go-datastore/query"
24 log "github.com/ipfs/go-log/v2"
25 "github.com/ipfs/kubo/config"
26 + "github.com/ipfs/kubo/core/shutdown"
27 "github.com/ipfs/kubo/repo"
28 "github.com/ipfs/kubo/repo/fsrepo"
29 irouting "github.com/ipfs/kubo/routing"
@@ -321,7 +322,7 @@ Learn more: https://github.com/ipfs/kubo/blob/master/docs/config.md#provide`,
322 }
323 lc.Append(fx.Hook{
324 OnStop: func(ctx context.Context) error {
324 - return sys.Close()
325 + return shutdown.CloseWithCtx(ctx, "legacy-provider", sys.Close)
326 },
327 })
328
@@ -745,6 +746,7 @@ func SweepingProviderOpt(cfg *config.Config) fx.Option {
746 ddhtprovider.WithMaxReprovideDelay(time.Hour),
747 ddhtprovider.WithOfflineDelay(cfg.Provide.DHT.OfflineDelay.WithDefault(config.DefaultProvideDHTOfflineDelay)),
748 ddhtprovider.WithConnectivityCheckOnlineInterval(1*time.Minute),
749 + ddhtprovider.WithSendProviderRecordTimeout(cfg.Provide.DHT.SendProviderRecordTimeout.WithDefault(config.DefaultProvideDHTSendProviderRecordTimeout)),
750
751 ddhtprovider.WithMaxWorkers(int(cfg.Provide.DHT.MaxWorkers.WithDefault(config.DefaultProvideDHTMaxWorkers))),
752 ddhtprovider.WithDedicatedPeriodicWorkers(int(cfg.Provide.DHT.DedicatedPeriodicWorkers.WithDefault(config.DefaultProvideDHTDedicatedPeriodicWorkers))),
@@ -790,6 +792,7 @@ func SweepingProviderOpt(cfg *config.Config) fx.Option {
792 dhtprovider.WithMaxReprovideDelay(time.Hour),
793 dhtprovider.WithOfflineDelay(cfg.Provide.DHT.OfflineDelay.WithDefault(config.DefaultProvideDHTOfflineDelay)),
794 dhtprovider.WithConnectivityCheckOnlineInterval(1 * time.Minute),
795 + dhtprovider.WithSendProviderRecordTimeout(cfg.Provide.DHT.SendProviderRecordTimeout.WithDefault(config.DefaultProvideDHTSendProviderRecordTimeout)),
796
797 dhtprovider.WithMaxWorkers(int(cfg.Provide.DHT.MaxWorkers.WithDefault(config.DefaultProvideDHTMaxWorkers))),
798 dhtprovider.WithDedicatedPeriodicWorkers(int(cfg.Provide.DHT.DedicatedPeriodicWorkers.WithDefault(config.DefaultProvideDHTDedicatedPeriodicWorkers))),
@@ -927,14 +930,15 @@ func SweepingProviderOpt(cfg *config.Config) fx.Option {
930
931 lc.Append(fx.Hook{
932 OnStop: func(ctx context.Context) error {
930 - // Close provider first - waits for all worker goroutines to exit.
931 - // This ensures no code can access keystore after this returns.
932 - if err := in.Provider.Close(); err != nil {
933 + // Close provider first; waits for all worker goroutines
934 + // to exit so nothing can access the keystore after this
935 + // returns. If ctx fires before provider drains, the
936 + // keystore close below sees an expired ctx and returns
937 + // immediately; the watchdog is the ultimate backstop.
938 + if err := shutdown.CloseWithCtx(ctx, "dht-provider", in.Provider.Close); err != nil {
939 providerLog.Errorw("error closing provider during shutdown", "error", err)
940 }
935 -
936 - // Close keystore - safe now, provider is fully shut down
937 - return in.Keystore.Close()
941 + return shutdown.CloseWithCtx(ctx, "keystore", in.Keystore.Close)
942 },
943 })
944 })
@@ -995,7 +999,16 @@ func SweepingProviderOpt(cfg *config.Config) fx.Option {
999 case <-ticker.C:
1000 }
1001
998 - stats := prov.Stats()
1002 + statsCtx, statsCancel := context.WithTimeout(gcCtx, time.Minute)
1003 + stats, err := prov.Stats(statsCtx)
1004 + statsCancel()
1005 + if err != nil {
1006 + if gcCtx.Err() != nil {
1007 + return
1008 + }
1009 + providerLog.Debugw("provider stats unavailable for reprovide alert", "err", err)
1010 + continue
1011 + }
1012 queuedWorkers = stats.Workers.QueuedPeriodic > 0
1013 queueSize = int64(stats.Queues.PendingRegionReprovides)
1014
core/shutdown/close.go new
+31
@@ -0,0 +1,31 @@
1 +package shutdown
2 +
3 +import (
4 + "context"
5 + "fmt"
6 + "time"
7 +
8 + logging "github.com/ipfs/go-log/v2"
9 +)
10 +
11 +var closeLog = logging.Logger("shutdown")
12 +
13 +// CloseWithCtx runs close in a goroutine and returns when it finishes or
14 +// when ctx is done, whichever comes first. If ctx fires before close
15 +// returns, the goroutine is leaked intentionally; the process is about to
16 +// exit, so the leak is bounded by process lifetime. Logs at ERROR which
17 +// subsystem failed to close in time so operators see it in journal/docker
18 +// logs.
19 +func CloseWithCtx(ctx context.Context, name string, close func() error) error {
20 + done := make(chan error, 1)
21 + start := time.Now()
22 + go func() { done <- close() }()
23 + select {
24 + case err := <-done:
25 + return err
26 + case <-ctx.Done():
27 + closeLog.Errorf("subsystem %q failed to close within shutdown deadline (after %s): %s",
28 + name, time.Since(start), ctx.Err())
29 + return fmt.Errorf("%s close: %w", name, ctx.Err())
30 + }
31 +}
core/shutdown/close_test.go new
+63
@@ -0,0 +1,63 @@
1 +package shutdown
2 +
3 +import (
4 + "context"
5 + "errors"
6 + "testing"
7 + "testing/synctest"
8 + "time"
9 +)
10 +
11 +const (
12 + // testFinishDeadline is the ctx deadline for the happy-path tests:
13 + // long enough that the close callback returns first.
14 + testFinishDeadline = time.Second
15 + // testTimeoutDeadline is the ctx deadline for the timeout test. Any
16 + // positive value works because the test runs under synctest's fake
17 + // clock; the choice only affects the exact-elapsed assertion below.
18 + testTimeoutDeadline = 50 * time.Millisecond
19 +)
20 +
21 +func TestCloseWithCtx_finishesBeforeDeadline(t *testing.T) {
22 + t.Parallel()
23 + ctx, cancel := context.WithTimeout(context.Background(), testFinishDeadline)
24 + defer cancel()
25 + if err := CloseWithCtx(ctx, "fast", func() error { return nil }); err != nil {
26 + t.Fatal(err)
27 + }
28 +}
29 +
30 +func TestCloseWithCtx_propagatesCloseError(t *testing.T) {
31 + t.Parallel()
32 + ctx, cancel := context.WithTimeout(context.Background(), testFinishDeadline)
33 + defer cancel()
34 + want := errors.New("close failed")
35 + err := CloseWithCtx(ctx, "bad", func() error { return want })
36 + if !errors.Is(err, want) {
37 + t.Fatalf("want %v, got %v", want, err)
38 + }
39 +}
40 +
41 +func TestCloseWithCtx_timesOut(t *testing.T) {
42 + synctest.Test(t, func(t *testing.T) {
43 + ctx, cancel := context.WithTimeout(context.Background(), testTimeoutDeadline)
44 + defer cancel()
45 + // release lets the simulated close exit after we've asserted on
46 + // CloseWithCtx. Without it, synctest panics with "blocked
47 + // goroutines remain" because production-side CloseWithCtx
48 + // intentionally leaks the goroutine when the deadline fires.
49 + release := make(chan struct{})
50 + start := time.Now()
51 + err := CloseWithCtx(ctx, "slow", func() error {
52 + <-release
53 + return nil
54 + })
55 + if elapsed := time.Since(start); elapsed != testTimeoutDeadline {
56 + t.Fatalf("want elapsed == %s, got %s", testTimeoutDeadline, elapsed)
57 + }
58 + if !errors.Is(err, context.DeadlineExceeded) {
59 + t.Fatalf("want DeadlineExceeded, got %v", err)
60 + }
61 + close(release)
62 + })
63 +}
core/shutdown/state.go new
+35
@@ -0,0 +1,35 @@
1 +// Package shutdown tracks daemon-wide graceful shutdown state. The daemon
2 +// command marks shutdown started when SIGTERM/SIGINT is received; the
3 +// "ipfs diag healthy" subcommand checks this state for Dockerfile
4 +// HEALTHCHECK and other monitoring.
5 +package shutdown
6 +
7 +import (
8 + "sync/atomic"
9 + "time"
10 +)
11 +
12 +// startedAt holds the unix-nano timestamp when shutdown began.
13 +// Zero means shutdown has not started.
14 +var startedAt atomic.Int64
15 +
16 +// MarkStarted records that graceful shutdown has begun. Safe to call
17 +// multiple times concurrently; only the first call wins. Returns true on
18 +// the first call, false on subsequent calls.
19 +func MarkStarted() bool {
20 + return startedAt.CompareAndSwap(0, time.Now().UnixNano())
21 +}
22 +
23 +// StartedAt returns when shutdown began, or the zero time if not started.
24 +func StartedAt() time.Time {
25 + n := startedAt.Load()
26 + if n == 0 {
27 + return time.Time{}
28 + }
29 + return time.Unix(0, n)
30 +}
31 +
32 +// InProgress reports whether shutdown has been initiated.
33 +func InProgress() bool {
34 + return startedAt.Load() != 0
35 +}
core/shutdown/state_test.go new
+77
@@ -0,0 +1,77 @@
1 +package shutdown
2 +
3 +import (
4 + "sync/atomic"
5 + "testing"
6 + "time"
7 +)
8 +
9 +// resetForTest clears the package-level state. Tests in this file mutate
10 +// global state, so they cannot run in parallel.
11 +func resetForTest(t *testing.T) {
12 + t.Helper()
13 + startedAt.Store(0)
14 +}
15 +
16 +func TestInProgressInitiallyFalse(t *testing.T) {
17 + resetForTest(t)
18 + if InProgress() {
19 + t.Fatal("InProgress() should be false before MarkStarted")
20 + }
21 + if !StartedAt().IsZero() {
22 + t.Fatal("StartedAt() should be zero time before MarkStarted")
23 + }
24 +}
25 +
26 +func TestMarkStartedFirstCallWins(t *testing.T) {
27 + resetForTest(t)
28 + if !MarkStarted() {
29 + t.Fatal("first MarkStarted() should return true")
30 + }
31 + if MarkStarted() {
32 + t.Fatal("second MarkStarted() should return false")
33 + }
34 + if !InProgress() {
35 + t.Fatal("InProgress() should be true after MarkStarted")
36 + }
37 + if StartedAt().IsZero() {
38 + t.Fatal("StartedAt() should be non-zero after MarkStarted")
39 + }
40 +}
41 +
42 +func TestMarkStartedPreservesFirstTimestamp(t *testing.T) {
43 + resetForTest(t)
44 + MarkStarted()
45 + first := StartedAt()
46 + // Sleep is intentional: it forces time.Now() to advance between the
47 + // two MarkStarted calls so a regression that replaces the CAS with a
48 + // plain Store would change StartedAt() and fail the assertion below.
49 + // Without the gap, both calls could land in the same nanosecond on
50 + // coarse-resolution clocks and mask the bug.
51 + time.Sleep(2 * time.Millisecond)
52 + MarkStarted() // second call must not overwrite
53 + if !StartedAt().Equal(first) {
54 + t.Fatalf("StartedAt() changed after second MarkStarted: %v != %v", StartedAt(), first)
55 + }
56 +}
57 +
58 +func TestMarkStartedConcurrent(t *testing.T) {
59 + resetForTest(t)
60 + const goroutines = 64
61 + var winners atomic.Int32
62 + done := make(chan struct{})
63 + for range goroutines {
64 + go func() {
65 + if MarkStarted() {
66 + winners.Add(1)
67 + }
68 + done <- struct{}{}
69 + }()
70 + }
71 + for range goroutines {
72 + <-done
73 + }
74 + if got := winners.Load(); got != 1 {
75 + t.Fatalf("expected exactly 1 winner across %d goroutines, got %d", goroutines, got)
76 + }
77 +}
docs/changelogs/v0.42.md
+15 -2
@@ -11,6 +11,7 @@ This release was brought to you by the [Shipyard](https://ipshipyard.com/) team.
11 - [Overview](#overview)
12 - [🔦 Highlights](#-highlights)
13 - [🐛 Fixed pin operations hanging under pinned reprovide strategies](#-fixed-pin-operations-hanging-under-pinned-reprovide-strategies)
14 + - [🐛 Reliable shutdown and container health checks](#-reliable-shutdown-and-container-health-checks)
15 - [🚨 ERROR log for listeners blocked by `Swarm.AddrFilters` or `Addresses.NoAnnounce`](#-error-log-for-listeners-blocked-by-swarmaddrfilters-or-addressesnoannounce)
16 - [📊 OpenTelemetry: scope info now exposed as labels](#-opentelemetry-scope-info-now-exposed-as-labels)
17 - [📦️ Dependency updates](#-dependency-updates)
@@ -27,7 +28,19 @@ This release was brought to you by the [Shipyard](https://ipshipyard.com/) team.
28
29 The pinner now snapshots the index under the read lock and releases it before the reprovider starts, so pin operations are no longer blocked by the reprovide cycle. The default `Provide.Strategy=all` was not affected.
30
30 -Daemon shutdown is also cleaner under these strategies: the pinner now cancels and drains in-flight work before the datastore closes, so the transient `pebble: closed` panic trace on exit is gone.
31 +#### 🐛 Reliable shutdown and container health checks
32 +
33 +Sending `SIGTERM` or `SIGINT` to kubo could leave the daemon stuck "half-shutdown": internal subsystems had stopped, but the process kept running and answering the RPC API. Docker and Kubernetes health checks reported the node as healthy while it had quietly stopped serving content. Recovery required a manual `docker restart`. Separately, the pinner could log a `pebble: closed` panic trace when the datastore closed before ongoing pin operations finished.
34 +
35 +What changed:
36 +
37 +- **Bounded shutdown.** A new [`Internal.ShutdownTimeout`](https://github.com/ipfs/kubo/blob/master/docs/config.md#internalshutdowntimeout) caps how long a stuck shutdown can run, so a zombie daemon recovers instead of staying half-alive. Routine shutdowns finish in seconds; this is a belt-and-suspenders ceiling against unknown bugs and future regressions. The 12-hour default is high enough that no real-world deployment hits it and low enough to recycle a stuck node well before its DHT provider records expire (22 hours). On expiry, the daemon logs which subsystem failed and exits with status `1`. Set `0` to disable.
38 +
39 +- **`ipfs diag healthy` subcommand.** Returns non-zero as soon as shutdown begins, even if the RPC API still answers. The kubo Docker image's `HEALTHCHECK` now uses it, so under `--restart=on-failure` or a Kubernetes liveness probe a half-shutdown daemon is recycled within seconds.
40 +
41 +- **Pinner shuts down cleanly.** The pinner cancels and waits for ongoing pin work before the datastore closes, removing the `pebble: closed` panic trace from shutdown logs.
42 +
43 +- **DHT provider deadlines.** `ipfs provide stat` now returns promptly when the caller cancels, instead of blocking on a slow keystore lookup (previously seen at over an hour). Each provider record sent to a peer is capped by [`Provide.DHT.SendProviderRecordTimeout`](https://github.com/ipfs/kubo/blob/master/docs/config.md#providedhtsendproviderrecordtimeout), so an unresponsive peer cannot stall a reprovide cycle.
44
45 #### 🚨 ERROR log for listeners blocked by `Swarm.AddrFilters` or `Addresses.NoAnnounce`
46
@@ -40,7 +53,7 @@ The Prometheus endpoint no longer emits the `otel_scope_info` metric. Each metri
53 #### 📦️ Dependency updates
54
55 - update `go-libp2p-pubsub` to [v0.16.0](https://github.com/libp2p/go-libp2p-pubsub/releases/tag/v0.16.0)
43 -- update `go-libp2p-kad-dht` to [v0.39.2](https://github.com/libp2p/go-libp2p-kad-dht/releases/tag/v0.39.2)
56 +- update `go-libp2p-kad-dht` to [b73e1e8](https://github.com/libp2p/go-libp2p-kad-dht/commit/b73e1e814f5f82e3554d350e61e33cae551084f6) (post-v0.39.2, includes [#1251](https://github.com/libp2p/go-libp2p-kad-dht/pull/1251) and [#1252](https://github.com/libp2p/go-libp2p-kad-dht/pull/1252))
57 - update `go-fuse/v2` to [v2.10.1](https://github.com/hanwen/go-fuse/releases/tag/v2.10.1)
58
59 ### 📝 Changelog
docs/config.md
+44
@@ -104,6 +104,7 @@ config file at runtime.
104 - [`Internal.Bitswap.BroadcastControl.MaxRandomPeers`](#internalbitswapbroadcastcontrolmaxrandompeers)
105 - [`Internal.Bitswap.BroadcastControl.SendToPendingPeers`](#internalbitswapbroadcastcontrolsendtopendingpeers)
106 - [`Internal.UnixFSShardingSizeThreshold`](#internalunixfsshardingsizethreshold)
107 + - [`Internal.ShutdownTimeout`](#internalshutdowntimeout)
108 - [`Ipns`](#ipns)
109 - [`Ipns.RepublishPeriod`](#ipnsrepublishperiod)
110 - [`Ipns.RecordLifetime`](#ipnsrecordlifetime)
@@ -144,6 +145,7 @@ config file at runtime.
145 - [`Provide.DHT.MaxProvideConnsPerWorker`](#providedhtmaxprovideconnsperworker)
146 - [`Provide.DHT.KeystoreBatchSize`](#providedhtkeystorebatchsize)
147 - [`Provide.DHT.OfflineDelay`](#providedhtofflinedelay)
148 + - [`Provide.DHT.SendProviderRecordTimeout`](#providedhtsendproviderrecordtimeout)
149 - [`Provide.BloomFPRate`](#providebloomfprate)
150 - [`Provider`](#provider)
151 - [`Provider.Enabled`](#providerenabled)
@@ -1909,6 +1911,28 @@ Type: `optionalInteger` (0 disables the limit, strongly discouraged)
1911 **Note:** This is an EXPERIMENTAL feature and may change or be removed in future releases.
1912 See [#10842](https://github.com/ipfs/kubo/issues/10842) for more information.
1913
1914 +### `Internal.ShutdownTimeout`
1915 +
1916 +Caps how long graceful shutdown is allowed to take. If `node.Close()` does
1917 +not return within this duration, the daemon logs which subsystem failed
1918 +and exits with status `1`. Set to `0` to wait forever (legacy behavior).
1919 +
1920 +The default `12h` guarantees the daemon cannot be stuck indefinitely on a
1921 +hung close hook, which matters for container orchestrators that otherwise
1922 +see a half-shutdown process as `healthy`. The value is smaller than the
1923 +22h DHT reprovide cycle, so a hung daemon recovers before missing more
1924 +than one cycle.
1925 +
1926 +Tune down for fast-restart environments. When tuning, raise the
1927 +orchestrator grace period (`--stop-timeout` for Docker,
1928 +`terminationGracePeriodSeconds` for Kubernetes) to at least this value so
1929 +the daemon exits gracefully before the orchestrator escalates to
1930 +`SIGKILL`.
1931 +
1932 +Default: `12h`
1933 +
1934 +Type: `optionalDuration` (`0` disables the cap)
1935 +
1936 ## `Ipns`
1937
1938 ### `Ipns.RepublishPeriod`
@@ -2640,6 +2664,26 @@ Default: `2h`
2664
2665 Type: `optionalDuration`
2666
2667 +#### `Provide.DHT.SendProviderRecordTimeout`
2668 +
2669 +Per-peer timeout applied to a single `ADD_PROVIDER` RPC sent during a provide
2670 +or reprovide operation. A peer that accepts the libp2p stream but never reads
2671 +the request can otherwise pin a provide worker goroutine until the connection
2672 +is dropped by the transport layer; this option bounds that wait.
2673 +
2674 +Healthy peers complete the round-trip in well under a second. The default
2675 +leaves significant headroom for slow links while keeping a hung peer from
2676 +stalling a worker.
2677 +
2678 +> [!NOTE]
2679 +> Lowering this value can speed up reprovide cycles when a non-trivial
2680 +> fraction of peers are slow or unresponsive, at the cost of giving up on
2681 +> genuinely slow but healthy peers.
2682 +
2683 +Default: `10s`
2684 +
2685 +Type: `optionalDuration` (positive)
2686 +
2687 ### `Provide.BloomFPRate`
2688
2689 Target false positive rate for the bloom filter used by the [`+unique` and
docs/examples/kubo-as-a-library/go.mod
+1 -1
@@ -118,7 +118,7 @@ require (
118 github.com/libp2p/go-doh-resolver v0.5.0 // indirect
119 github.com/libp2p/go-flow-metrics v0.3.0 // indirect
120 github.com/libp2p/go-libp2p-asn-util v0.4.1 // indirect
121 - github.com/libp2p/go-libp2p-kad-dht v0.39.2 // indirect
121 + github.com/libp2p/go-libp2p-kad-dht v0.39.3-0.20260513140308-b73e1e814f5f // indirect
122 github.com/libp2p/go-libp2p-kbucket v0.8.0 // indirect
123 github.com/libp2p/go-libp2p-pubsub v0.16.0 // indirect
124 github.com/libp2p/go-libp2p-pubsub-router v0.6.0 // indirect
docs/examples/kubo-as-a-library/go.sum
+2 -2
@@ -494,8 +494,8 @@ github.com/libp2p/go-libp2p-asn-util v0.4.1 h1:xqL7++IKD9TBFMgnLPZR6/6iYhawHKHl9
494 github.com/libp2p/go-libp2p-asn-util v0.4.1/go.mod h1:d/NI6XZ9qxw67b4e+NgpQexCIiFYJjErASrYW4PFDN8=
495 github.com/libp2p/go-libp2p-core v0.2.4/go.mod h1:STh4fdfa5vDYr0/SzYYeqnt+E6KfEV5VxfIrm0bcI0g=
496 github.com/libp2p/go-libp2p-core v0.3.0/go.mod h1:ACp3DmS3/N64c2jDzcV429ukDpicbL6+TrrxANBjPGw=
497 -github.com/libp2p/go-libp2p-kad-dht v0.39.2 h1:L0VVfNwZnlyfS56lgtUbfevx1La9GGiEnifHndLaA40=
498 -github.com/libp2p/go-libp2p-kad-dht v0.39.2/go.mod h1:DwCTwO3ZhBC3sGsAEG78D2LlAY0q00/HUHx8mwm/gYM=
497 +github.com/libp2p/go-libp2p-kad-dht v0.39.3-0.20260513140308-b73e1e814f5f h1:PH/bOCidYs4UHu3EtHxqQvxMyWHMMXeSJUlM4wcpK28=
498 +github.com/libp2p/go-libp2p-kad-dht v0.39.3-0.20260513140308-b73e1e814f5f/go.mod h1:iLUjII47u3/HjxyhucI2lhsl29lrzlAs/ym16+H40jE=
499 github.com/libp2p/go-libp2p-kbucket v0.3.1/go.mod h1:oyjT5O7tS9CQurok++ERgc46YLwEpuGoFq9ubvoUOio=
500 github.com/libp2p/go-libp2p-kbucket v0.8.0 h1:QAK7RzKJpYe+EuSEATAaaHYMYLkPDGC18m9jxPLnU8s=
501 github.com/libp2p/go-libp2p-kbucket v0.8.0/go.mod h1:JMlxqcEyKwO6ox716eyC0hmiduSWZZl6JY93mGaaqc4=
go.mod
+1 -1
@@ -52,7 +52,7 @@ require (
52 github.com/libp2p/go-doh-resolver v0.5.0
53 github.com/libp2p/go-libp2p v0.48.0
54 github.com/libp2p/go-libp2p-http v0.5.0
55 - github.com/libp2p/go-libp2p-kad-dht v0.39.2
55 + github.com/libp2p/go-libp2p-kad-dht v0.39.3-0.20260513140308-b73e1e814f5f
56 github.com/libp2p/go-libp2p-kbucket v0.8.0
57 github.com/libp2p/go-libp2p-pubsub v0.16.0
58 github.com/libp2p/go-libp2p-pubsub-router v0.6.0
go.sum
+2 -2
@@ -549,8 +549,8 @@ github.com/libp2p/go-libp2p-gostream v0.6.0 h1:QfAiWeQRce6pqnYfmIVWJFXNdDyfiR/qk
549 github.com/libp2p/go-libp2p-gostream v0.6.0/go.mod h1:Nywu0gYZwfj7Jc91PQvbGU8dIpqbQQkjWgDuOrFaRdA=
550 github.com/libp2p/go-libp2p-http v0.5.0 h1:+x0AbLaUuLBArHubbbNRTsgWz0RjNTy6DJLOxQ3/QBc=
551 github.com/libp2p/go-libp2p-http v0.5.0/go.mod h1:glh87nZ35XCQyFsdzZps6+F4HYI6DctVFY5u1fehwSg=
552 -github.com/libp2p/go-libp2p-kad-dht v0.39.2 h1:L0VVfNwZnlyfS56lgtUbfevx1La9GGiEnifHndLaA40=
553 -github.com/libp2p/go-libp2p-kad-dht v0.39.2/go.mod h1:DwCTwO3ZhBC3sGsAEG78D2LlAY0q00/HUHx8mwm/gYM=
552 +github.com/libp2p/go-libp2p-kad-dht v0.39.3-0.20260513140308-b73e1e814f5f h1:PH/bOCidYs4UHu3EtHxqQvxMyWHMMXeSJUlM4wcpK28=
553 +github.com/libp2p/go-libp2p-kad-dht v0.39.3-0.20260513140308-b73e1e814f5f/go.mod h1:iLUjII47u3/HjxyhucI2lhsl29lrzlAs/ym16+H40jE=
554 github.com/libp2p/go-libp2p-kbucket v0.3.1/go.mod h1:oyjT5O7tS9CQurok++ERgc46YLwEpuGoFq9ubvoUOio=
555 github.com/libp2p/go-libp2p-kbucket v0.8.0 h1:QAK7RzKJpYe+EuSEATAaaHYMYLkPDGC18m9jxPLnU8s=
556 github.com/libp2p/go-libp2p-kbucket v0.8.0/go.mod h1:JMlxqcEyKwO6ox716eyC0hmiduSWZZl6JY93mGaaqc4=
test/cli/shutdown_timeout_test.go new
+74
@@ -0,0 +1,74 @@
1 +package cli
2 +
3 +import (
4 + "testing"
5 + "time"
6 +
7 + "github.com/ipfs/kubo/config"
8 + "github.com/ipfs/kubo/test/cli/harness"
9 + "github.com/stretchr/testify/require"
10 +)
11 +
12 +const (
13 + // testShutdownTimeout overrides DefaultShutdownTimeout so the test
14 + // runs in seconds rather than the production default.
15 + testShutdownTimeout = 10 * time.Second
16 + // testShutdownCompletionBound is a soft upper bound for StopDaemon in
17 + // this test. StopDaemon escalates SIGTERM, SIGTERM, SIGQUIT, SIGKILL
18 + // itself (see harness/node.go), so anything close to this bound
19 + // indicates kubo's own bounded-shutdown logic failed.
20 + testShutdownCompletionBound = testShutdownTimeout + 5*time.Second
21 +)
22 +
23 +// TestShutdownTimeoutHonored exercises the bounded-shutdown logic end-to-end
24 +// for the common case (no hung subsystems): the daemon must shut down
25 +// cleanly well within the configured ShutdownTimeout, and pinned/MFS data
26 +// must survive across the restart.
27 +func TestShutdownTimeoutHonored(t *testing.T) {
28 + t.Parallel()
29 + h := harness.NewT(t)
30 + node := h.NewNode().Init()
31 + node.UpdateConfig(func(cfg *config.Config) {
32 + cfg.Internal.ShutdownTimeout = config.NewOptionalDuration(testShutdownTimeout)
33 + })
34 + node.StartDaemon()
35 +
36 + // Real data-path work that must survive shutdown.
37 + addCID := node.PipeStrToIPFS("survives shutdown", "add", "-q").Stdout.Trimmed()
38 + node.IPFS("files", "mkdir", "/persisted")
39 +
40 + // "diag healthy" must succeed while the daemon is running normally.
41 + require.Equal(t, 0, node.RunIPFS("diag", "healthy").ExitCode(),
42 + "diag healthy should succeed before shutdown is initiated")
43 +
44 + start := time.Now()
45 + node.StopDaemon()
46 + require.Less(t, time.Since(start), testShutdownCompletionBound,
47 + "graceful shutdown should complete well within the configured ShutdownTimeout")
48 +
49 + // Restart and verify data survived.
50 + node.StartDaemon()
51 + require.Contains(t, node.IPFS("pin", "ls").Stdout.String(), addCID,
52 + "pinned CID should survive shutdown+restart")
53 + require.Contains(t, node.IPFS("files", "ls").Stdout.String(), "persisted",
54 + "MFS content should survive shutdown+restart")
55 +}
56 +
57 +// TestShutdownTimeoutDisabled verifies that ShutdownTimeout=0 opts out of
58 +// the bounded-shutdown logic and behaves like legacy kubo (no watchdog,
59 +// no app.Stop deadline). The daemon must still shut down cleanly because
60 +// no subsystem is actually hung.
61 +func TestShutdownTimeoutDisabled(t *testing.T) {
62 + t.Parallel()
63 + h := harness.NewT(t)
64 + node := h.NewNode().Init()
65 + node.UpdateConfig(func(cfg *config.Config) {
66 + cfg.Internal.ShutdownTimeout = config.NewOptionalDuration(0)
67 + })
68 + node.StartDaemon()
69 +
70 + start := time.Now()
71 + node.StopDaemon()
72 + require.Less(t, time.Since(start), testShutdownCompletionBound,
73 + "graceful shutdown should still complete in reasonable time with ShutdownTimeout=0")
74 +}
test/dependencies/go.mod
+1 -1
@@ -183,7 +183,7 @@ require (
183 github.com/libp2p/go-flow-metrics v0.3.0 // indirect
184 github.com/libp2p/go-libp2p v0.48.0 // indirect
185 github.com/libp2p/go-libp2p-asn-util v0.4.1 // indirect
186 - github.com/libp2p/go-libp2p-kad-dht v0.39.2 // indirect
186 + github.com/libp2p/go-libp2p-kad-dht v0.39.3-0.20260513140308-b73e1e814f5f // indirect
187 github.com/libp2p/go-libp2p-kbucket v0.8.0 // indirect
188 github.com/libp2p/go-libp2p-record v0.3.1 // indirect
189 github.com/libp2p/go-libp2p-routing-helpers v0.7.5 // indirect
test/dependencies/go.sum
+2 -2
@@ -582,8 +582,8 @@ github.com/libp2p/go-libp2p v0.48.0 h1:h2BrLAgrj7X8bEN05K7qmrjpNHYA+6tnsGRdprjTn
582 github.com/libp2p/go-libp2p v0.48.0/go.mod h1:Q1fBZNdmC2Hf82husCTfkKJVfHm2we5zk+NWmOGEmWk=
583 github.com/libp2p/go-libp2p-asn-util v0.4.1 h1:xqL7++IKD9TBFMgnLPZR6/6iYhawHKHl950SO9L6n94=
584 github.com/libp2p/go-libp2p-asn-util v0.4.1/go.mod h1:d/NI6XZ9qxw67b4e+NgpQexCIiFYJjErASrYW4PFDN8=
585 -github.com/libp2p/go-libp2p-kad-dht v0.39.2 h1:L0VVfNwZnlyfS56lgtUbfevx1La9GGiEnifHndLaA40=
586 -github.com/libp2p/go-libp2p-kad-dht v0.39.2/go.mod h1:DwCTwO3ZhBC3sGsAEG78D2LlAY0q00/HUHx8mwm/gYM=
585 +github.com/libp2p/go-libp2p-kad-dht v0.39.3-0.20260513140308-b73e1e814f5f h1:PH/bOCidYs4UHu3EtHxqQvxMyWHMMXeSJUlM4wcpK28=
586 +github.com/libp2p/go-libp2p-kad-dht v0.39.3-0.20260513140308-b73e1e814f5f/go.mod h1:iLUjII47u3/HjxyhucI2lhsl29lrzlAs/ym16+H40jE=
587 github.com/libp2p/go-libp2p-kbucket v0.8.0 h1:QAK7RzKJpYe+EuSEATAaaHYMYLkPDGC18m9jxPLnU8s=
588 github.com/libp2p/go-libp2p-kbucket v0.8.0/go.mod h1:JMlxqcEyKwO6ox716eyC0hmiduSWZZl6JY93mGaaqc4=
589 github.com/libp2p/go-libp2p-record v0.3.1 h1:cly48Xi5GjNw5Wq+7gmjfBiG9HCzQVkiZOUZ8kUl+Fg=