cmd/files: flush parent folders (#10630)
* cmd/files: flush parent folders This is a mitigation to increased MFS memory usage in the course of many writes operations. The underlying issue is the unbounded growth of the mfs directory cache in boxo. In the latest boxo version, this cache can be cleared by calling Flush() on the folder. In order to trigger that, we call Flush() on the parent folder of the file/folder where the write-operations are happening. To flushing the parent folder allows it to grow unbounded. Then, any read operation to that folder or parents (i.e. stat), will trigger a sync-operation to match the cache to the underlying unixfs structure (and obtain the correct node-cid). This sync operation must visit every item in the cache. When the cache has grown too much, and the underlying unixfs-folder has switched into a HAMT, the operation can take minutes. Thus, we should clear the cache often and the Flush flag is a good indicator that we can let it go. Users can always run with --flush=false and flush at regular intervals during their MFS writes if they want to extract some performance. Fixes #8694, #10588. * cmd/files: docs and changelog for --flush changes
Hector Sanjuan committed
Dec 19, 2024 at 23:07 UTC
ecb25581c64a79432a48eb65ccf4f8c90e8518ed
2 files changed
+68
-18
core/commands/files.go
+64
-18
@@ -59,16 +59,18 @@ Content added with "ipfs add" (which by default also becomes pinned), is not
59
added to MFS. Any content can be lazily referenced from MFS with the command
60
"ipfs files cp /ipfs/<cid> /some/path/" (see ipfs files cp --help).
61
62
-
63
-NOTE:
64
-Most of the subcommands of 'ipfs files' accept the '--flush' flag. It defaults
65
-to true. Use caution when setting this flag to false. It will improve
62
+NOTE: Most of the subcommands of 'ipfs files' accept the '--flush' flag. It
63
+defaults to true and ensures two things: 1) that the changes are reflected in
64
+the full MFS structure (updated CIDs) 2) that the parent-folder's cache is
65
+cleared. Use caution when setting this flag to false. It will improve
66
performance for large numbers of file operations, but it does so at the cost
67
-of consistency guarantees. If the daemon is unexpectedly killed before running
68
-'ipfs files flush' on the files in question, then data may be lost. This also
69
-applies to run 'ipfs repo gc' concurrently with '--flush=false'
70
-operations.
71
-`,
67
+of consistency guarantees and unbound growth of the directories' in-memory
68
+caches. If the daemon is unexpectedly killed before running 'ipfs files
69
+flush' on the files in question, then data may be lost. This also applies to
70
+run 'ipfs repo gc' concurrently with '--flush=false' operations. We recommend
71
+flushing paths reguarly with 'ipfs files flush', specially the folders on
72
+which many write operations are happening, as a way to clear the directory
73
+cache, free memory and speed up read operations.`,
74
},
75
Options: []cmds.Option{
76
cmds.BoolOption(filesFlushOptionName, "f", "Flush target and ancestors after write.").WithDefault(true),
@@ -491,10 +493,14 @@ being GC'ed.
493
}
494
495
if flush {
494
- _, err := mfs.FlushPath(req.Context, nd.FilesRoot, dst)
495
- if err != nil {
496
+ if _, err := mfs.FlushPath(req.Context, nd.FilesRoot, dst); err != nil {
497
return fmt.Errorf("cp: cannot flush the created file %s: %s", dst, err)
498
}
499
+ // Flush parent to clear directory cache and free memory.
500
+ parent := gopath.Dir(dst)
501
+ if _, err = mfs.FlushPath(req.Context, nd.FilesRoot, parent); err != nil {
502
+ return fmt.Errorf("cp: cannot flush the created file's parent folder %s: %s", dst, err)
503
+ }
504
}
505
506
return nil
@@ -792,10 +798,30 @@ Example:
798
}
799
800
err = mfs.Mv(nd.FilesRoot, src, dst)
795
- if err == nil && flush {
796
- _, err = mfs.FlushPath(req.Context, nd.FilesRoot, "/")
801
+ if err != nil {
802
+ return err
803
}
798
- return err
804
+ if flush {
805
+ parentSrc := gopath.Dir(src)
806
+ parentDst := gopath.Dir(dst)
807
+ // Flush parent to clear directory cache and free memory.
808
+ if _, err = mfs.FlushPath(req.Context, nd.FilesRoot, parentDst); err != nil {
809
+ return fmt.Errorf("cp: cannot flush the destination file's parent folder %s: %s", dst, err)
810
+ }
811
+
812
+ // Avoid re-flushing when moving within the same folder.
813
+ if parentSrc != parentDst {
814
+ if _, err = mfs.FlushPath(req.Context, nd.FilesRoot, parentSrc); err != nil {
815
+ return fmt.Errorf("cp: cannot flush the source's file's parent folder %s: %s", dst, err)
816
+ }
817
+ }
818
+
819
+ if _, err = mfs.FlushPath(req.Context, nd.FilesRoot, "/"); err != nil {
820
+ return err
821
+ }
822
+ }
823
+
824
+ return nil
825
},
826
}
827
@@ -943,6 +969,17 @@ See '--to-files' in 'ipfs add --help' for more information.
969
flog.Error("files: error closing file mfs file descriptor", err)
970
}
971
}
972
+ if flush {
973
+ // Flush parent to clear directory cache and free memory.
974
+ parent := gopath.Dir(path)
975
+ if _, err := mfs.FlushPath(req.Context, nd.FilesRoot, parent); err != nil {
976
+ if retErr == nil {
977
+ retErr = err
978
+ } else {
979
+ flog.Error("files: flushing the parent folder", err)
980
+ }
981
+ }
982
+ }
983
}()
984
985
if trunc {
@@ -1105,11 +1142,20 @@ Change the CID version or hash function of the root node of a given path.
1142
return err
1143
}
1144
1108
- err = updatePath(nd.FilesRoot, path, prefix)
1109
- if err == nil && flush {
1110
- _, err = mfs.FlushPath(req.Context, nd.FilesRoot, path)
1145
+ if err := updatePath(nd.FilesRoot, path, prefix); err != nil {
1146
+ return err
1147
}
1112
- return err
1148
+ if flush {
1149
+ if _, err = mfs.FlushPath(req.Context, nd.FilesRoot, path); err != nil {
1150
+ return err
1151
+ }
1152
+ // Flush parent to clear directory cache and free memory.
1153
+ parent := gopath.Dir(path)
1154
+ if _, err = mfs.FlushPath(req.Context, nd.FilesRoot, parent); err != nil {
1155
+ return err
1156
+ }
1157
+ }
1158
+ return nil
1159
},
1160
}
1161
docs/changelogs/v0.33.md
+4
@@ -31,6 +31,10 @@ If you depended on removed ones, please fill an issue to add them to the upstrea
31
32
Onboarding files and directories with `ipfs add --to-files` now requires non-empty names. due to this, The `--to-files` and `--wrap` options are now mutually exclusive ([#10612](https://github.com/ipfs/kubo/issues/10612)).
33
34
+#### MFS stability with large number of writes
35
+
36
+We have fixed a number of issues that were triggered by writing or copying many files onto an MFS folder: increased memory usage first, then CPU, disk usage, and eventually a deadlock on write operations. The details of the fixes can be read at [#10630](https://github.com/ipfs/kubo/pull/10630) and [#10623](https://github.com/ipfs/kubo/pull/10623). The result is that writing large amounts of files to an MFS folder should now be possible without major issues. It is possible, as before, to speed up the operations using the `ipfs files --flush=false <op> ...` flag, but it is recommended to switch to `ipfs files --flush=true <op> ...` regularly, or call `ipfs files flush` on the working directory regularly, as this will flush, clear the directory cache and speed up reads.
37
+
38
#### 📦️ Dependency updates
39
40
- update `boxo` to [v0.26.0](https://github.com/ipfs/boxo/releases/tag/v0.26.0)