feat: bound graceful shutdown, add diag healthy (#11329)
* feat: bound graceful shutdown, add diag healthy Replace unbounded app.Stop(context.Background()) with a deadline-bounded context driven by a new Internal.ShutdownTimeout config (default 12h, 0 disables). Add an os.Exit(1) watchdog at the same deadline so an FX OnStop hook that never returns can no longer hang the daemon. Add ipfs diag healthy: fails when shutdown has been initiated or when the DAG pipeline cannot resolve the well-known empty-directory CID. Dockerfile HEALTHCHECK now uses it so orchestrators recycle half- shutdown daemons. - core/shutdown: new pkg; atomic startedAt + CloseWithCtx helper - core/builder.go: app.Stop bounded by ShutdownTimeout - cmd/ipfs/kubo/daemon.go: watchdog + MarkStarted on signal - core/commands/diag.go: new healthy subcommand - core/node/{bitswap,libp2p/host,libp2p/routing}.go: OnStop hooks wrapped - config/internal.go: ShutdownTimeout + DefaultShutdownTimeout=12h - Dockerfile: HEALTHCHECK uses "ipfs diag healthy" - docs/{config,changelogs/v0.42}.md: documented - test/cli: enabled + disabled path tests * feat: bound provider stats and ADD_PROVIDER sends bumps go-libp2p-kad-dht past v0.39.2 to b73e1e8 to pick up two related provider bug fixes. - ipfs provide stat now honors client cancellation and deadlines instead of blocking indefinitely behind a slow keystore lookup - adds Provide.DHT.SendProviderRecordTimeout capping each ADD_PROVIDER RPC so unresponsive peers cannot pin a provide worker and stall reprovide cycles - internal reprovide-alert poller bounds its Stats call so a hung keystore.Size cannot delay shutdown * test(shutdown): use synctest for timeout test, document sleep CloseWithCtx_timesOut now runs in a synctest bubble so the deadline assertion is exact (no wall-clock slack), and the simulated close uses a release channel to drain the bubble cleanly after the leak point. The two happy-path tests stay unchanged because their close funcs return immediately and gain nothing from a fake clock. Comment the 2ms sleep in TestMarkStartedPreservesFirstTimestamp so its role (forcing time.Now() to advance between the two MarkStarted calls so a CAS to Store regression is detectable) is not lost. Addresses ipfs/kubo#11329 (review). * fix(pinner): bound pinner Close with shutdown deadline The boxo Pinner.Close contract notes that an in-flight op ignoring its ctx (a downstream bug) can block Close, so the host must bound it at the call site. Wrapping the OnStop hook with CloseWithCtx honors Internal.ShutdownTimeout and surfaces an actionable "subsystem 'pinner' failed to close" log on hang instead of leaving only the watchdog os.Exit(1) trace. * fix(shutdown): bound remaining I/O-touching OnStop hooks Wrap the OnStop hooks whose Close can plausibly block on disk or network: repo (datastore flush + lock release), mfs-root (datastore writes via DAGService), peering (waits on libp2p peer goroutines), legacy-provider (in-flight reprovide RPCs), and the dht-provider plus keystore pair under SweepingProvider. In-memory closes (blockservice, peerstore, resource-manager) are left as-is since they cannot realistically hang. For the dht-provider/keystore pair, provider closes first so nothing can access the keystore afterwards. If the shutdown ctx fires mid-provider-drain, the keystore close sees an expired ctx and returns immediately; the watchdog os.Exit(1) is the ultimate backstop, and keystore writes are fsync'd on put so missing the explicit close is recoverable on next boot. * fix(shutdown): bound remaining in-memory OnStop hooks Wrap blockservice, peerstore, and resource-manager Close hooks with CloseWithCtx for uniformity. These are pure in-memory operations unlikely to hang in practice, but wrapping costs nothing and makes the shutdown audit trail uniform: every OnStop hook now honors the deadline and surfaces a named subsystem on timeout. * fix(shutdown): bound autoRelayFeeder OnStop on ctx OnStop waited on the feeder goroutine via <-done without honoring the shutdown ctx. The goroutine itself selects on ctx in every loop case, so cancel() normally suffices, but a stuck downstream dht.WAN.GetClosestPeers that ignored its ctx could block fx.Stop indefinitely. Adding the ctx.Done() select case mirrors the reprovideAlert pattern in provider.go and lets the shutdown deadline reclaim control even with a misbehaving DHT. * docs(changelog): merge shutdown entries into one user-facing section Combine the pinner-on-shutdown paragraph with the bounded-shutdown section under a single "Reliable shutdown and container health checks" heading. Lead with the visible symptoms (half-shutdown daemons, healthy-but-dead container reports, manual docker restart) instead of fx OnStop jargon. Frame Internal.ShutdownTimeout as a belt-and-suspenders ceiling, with the 12-hour default sized against the 22-hour DHT provider record expiration.