45e8b7c2d4 · matheusbsbs/matheus-bagliano

@@ -14,37 +14,41 @@ Table of contents:

14 * Reference Emails

15

16

17 -=== Terminology ===

17 +== Terminology ==

18

19 -cone mode: one of two modes for specifying the desired subset of files

19 +*`cone mode`*::

20 + one of two modes for specifying the desired subset of files

21 in a sparse-checkout. In cone-mode, the user specifies

22 directories (getting both everything under that directory as

23 well as everything in leading directories), while in non-cone

24 mode, the user specifies gitignore-style patterns. Controlled

25 by the --[no-]cone option to sparse-checkout init|set.

26

26 -SKIP_WORKTREE: When tracked files do not match the sparse specification and

27 +*`SKIP_WORKTREE`*::

28 + When tracked files do not match the sparse specification and

29 are removed from the working tree, the file in the index is marked

30 with a SKIP_WORKTREE bit. Note that if a tracked file has the

31 SKIP_WORKTREE bit set but the file is later written by the user to

32 the working tree anyway, the SKIP_WORKTREE bit will be cleared at

33 the beginning of any subsequent Git operation.

32 -

33 - Most sparse checkout users are unaware of this implementation

34 - detail, and the term should generally be avoided in user-facing

35 - descriptions and command flags. Unfortunately, prior to the

36 - `sparse-checkout` subcommand this low-level detail was exposed,

37 - and as of time of writing, is still exposed in various places.

38 -

39 -sparse-checkout: a subcommand in git used to reduce the files present in

34 ++

35 +Most sparse checkout users are unaware of this implementation

36 +detail, and the term should generally be avoided in user-facing

37 +descriptions and command flags. Unfortunately, prior to the

38 +`sparse-checkout` subcommand this low-level detail was exposed,

39 +and as of time of writing, is still exposed in various places.

40 +

41 +*`sparse-checkout`*::

42 + a subcommand in git used to reduce the files present in

43 the working tree to a subset of all tracked files. Also, the

44 name of the file in the $GIT_DIR/info directory used to track

45 the sparsity patterns corresponding to the user's desired

46 subset.

47

45 -sparse cone: see cone mode

48 +*`sparse cone`*:: see cone mode

49

47 -sparse directory: An entry in the index corresponding to a directory, which

50 +*`sparse directory`*::

51 + An entry in the index corresponding to a directory, which

52 appears in the index instead of all the files under that directory

53 that would normally appear. See also sparse-index. Something that

54 can cause confusion is that the "sparse directory" does NOT match

@@ -52,7 +56,8 @@ sparse directory: An entry in the index corresponding to a directory, which

56 working tree. May be renamed in the future (e.g. to "skipped

57 directory").

58

55 -sparse index: A special mode for sparse-checkout that also makes the

59 +*`sparse index`*::

60 + A special mode for sparse-checkout that also makes the

61 index sparse by recording a directory entry in lieu of all the

62 files underneath that directory (thus making that a "skipped

63 directory" which unfortunately has also been called a "sparse

@@ -60,7 +65,8 @@ sparse index: A special mode for sparse-checkout that also makes the

65 directories. Controlled by the --[no-]sparse-index option to

66 init|set|reapply.

67

63 -sparsity patterns: patterns from $GIT_DIR/info/sparse-checkout used to

68 +*`sparsity patterns`*::

69 + patterns from $GIT_DIR/info/sparse-checkout used to

70 define the set of files of interest. A warning: It is easy to

71 over-use this term (or the shortened "patterns" term), for two

72 reasons: (1) users in cone mode specify directories rather than

@@ -70,7 +76,8 @@ sparsity patterns: patterns from $GIT_DIR/info/sparse-checkout used to

76 transiently differ in the working tree or index from the sparsity

77 patterns (see "Sparse specification vs. sparsity patterns").

78

73 -sparse specification: The set of paths in the user's area of focus. This

79 +*`sparse specification`*::

80 + The set of paths in the user's area of focus. This

81 is typically just the tracked files that match the sparsity

82 patterns, but the sparse specification can temporarily differ and

83 include additional files. (See also "Sparse specification

@@ -87,12 +94,13 @@ sparse specification: The set of paths in the user's area of focus. This

94 * If working with the index and the working copy, the sparse

95 specification is the union of the paths from above.

96

90 -vivifying: When a command restores a tracked file to the working tree (and

97 +*`vivifying`*::

98 + When a command restores a tracked file to the working tree (and

99 hopefully also clears the SKIP_WORKTREE bit in the index for that

100 file), this is referred to as "vivifying" the file.

101

102

95 -=== Purpose of sparse-checkouts ===

103 +== Purpose of sparse-checkouts ==

104

105 sparse-checkouts exist to allow users to work with a subset of their

106 files.

@@ -120,14 +128,12 @@ those usecases, sparse-checkouts can modify different subcommands in over a

128 half dozen different ways. Let's start by considering the high level

129 usecases:

130

123 - A) Users are _only_ interested in the sparse portion of the repo

124 -

125 - A*) Users are _only_ interested in the sparse portion of the repo

126 - that they have downloaded so far

127 -

128 - B) Users want a sparse working tree, but are working in a larger whole

129 -

130 - C) sparse-checkout is a behind-the-scenes implementation detail allowing

131 +[horizontal]

132 +A):: Users are _only_ interested in the sparse portion of the repo

133 +A*):: Users are _only_ interested in the sparse portion of the repo

134 + that they have downloaded so far

135 +B):: Users want a sparse working tree, but are working in a larger whole

136 +C):: sparse-checkout is a behind-the-scenes implementation detail allowing

137 Git to work with a specially crafted in-house virtual file system;

138 users are actually working with a "full" working tree that is

139 lazily populated, and sparse-checkout helps with the lazy population

@@ -136,7 +142,7 @@ usecases:

142 It may be worth explaining each of these in a bit more detail:

143

144

139 - (Behavior A) Users are _only_ interested in the sparse portion of the repo

145 +=== (Behavior A) Users are _only_ interested in the sparse portion of the repo

146

147 These folks might know there are other things in the repository, but

148 don't care. They are uninterested in other parts of the repository, and

@@ -163,8 +169,7 @@ side-effects of various other commands (such as the printed diffstat

169 after a merge or pull) can lead to worries about local repository size

170 growing unnecessarily[10].

171

166 - (Behavior A*) Users are _only_ interested in the sparse portion of the repo

167 - that they have downloaded so far (a variant on the first usecase)

172 +=== (Behavior A*) Users are _only_ interested in the sparse portion of the repo that they have downloaded so far (a variant on the first usecase)

173

174 This variant is driven by folks who using partial clones together with

175 sparse checkouts and do disconnected development (so far sounding like a

@@ -173,15 +178,14 @@ reason for yet another variant is that downloading even just the blobs

178 through history within their sparse specification may be too much, so they

179 only download some. They would still like operations to succeed without

180 network connectivity, though, so things like `git log -S${SEARCH_TERM} -p`

176 -or `git grep ${SEARCH_TERM} OLDREV ` would need to be prepared to provide

181 +or `git grep ${SEARCH_TERM} OLDREV` would need to be prepared to provide

182 partial results that depend on what happens to have been downloaded.

183

184 This variant could be viewed as Behavior A with the sparse specification

185 for history querying operations modified from "sparsity patterns" to

186 "sparsity patterns limited to the blobs we have already downloaded".

187

183 - (Behavior B) Users want a sparse working tree, but are working in a

184 - larger whole

188 +=== (Behavior B) Users want a sparse working tree, but are working in a larger whole

189

190 Stolee described this usecase this way[11]:

191

@@ -229,8 +233,7 @@ those expensive checks when interacting with the working copy, and may

233 prefer getting "unrelated" results from their history queries over having

234 slow commands.

235

232 - (Behavior C) sparse-checkout is an implementational detail supporting a

233 - special VFS.

236 +=== (Behavior C) sparse-checkout is an implementational detail supporting a special VFS.

237

238 This usecase goes slightly against the traditional definition of

239 sparse-checkout in that it actually tries to present a full or dense

@@ -255,13 +258,13 @@ will perceive the checkout as dense, and commands should thus behave as if

258 all files are present.

259

260

258 -=== Usecases of primary concern ===

261 +== Usecases of primary concern ==

262

263 Most of the rest of this document will focus on Behavior A and Behavior

264 B. Some notes about the other two cases and why we are not focusing on

265 them:

266

264 - (Behavior A*)

267 +=== (Behavior A*)

268

269 Supporting this usecase is estimated to be difficult and a lot of work.

270 There are no plans to implement it currently, but it may be a potential

@@ -275,7 +278,7 @@ valid for this usecase, with the only exception being that it redefines the

278 sparse specification to restrict it to already-downloaded blobs. The hard

279 part is in making commands capable of respecting that modified definition.

280

278 - (Behavior C)

281 +=== (Behavior C)

282

283 This usecase violates some of the early sparse-checkout documented

284 assumptions (since files marked as SKIP_WORKTREE will be displayed to users

@@ -300,20 +303,20 @@ Behavior C do not assume they are part of the Behavior B camp and propose

303 patches that break things for the real Behavior B folks.

304

305

303 -=== Oversimplified mental models ===

306 +== Oversimplified mental models ==

307

308 An oversimplification of the differences in the above behaviors is:

309

307 - Behavior A: Restrict worktree and history operations to sparse specification

308 - Behavior B: Restrict worktree operations to sparse specification; have any

309 - history operations work across all files

310 - Behavior C: Do not restrict either worktree or history operations to the

311 - sparse specification...with the exception of branch checkouts or

312 - switches which avoid writing files that will match the index so

313 - they can later lazily be populated instead.

310 +(Behavior A):: Restrict worktree and history operations to sparse specification

311 +(Behavior B):: Restrict worktree operations to sparse specification; have any

312 + history operations work across all files

313 +(Behavior C):: Do not restrict either worktree or history operations to the

314 + sparse specification...with the exception of branch checkouts or

315 + switches which avoid writing files that will match the index so

316 + they can later lazily be populated instead.

317

318

316 -=== Desired behavior ===

319 +== Desired behavior ==

320

321 As noted previously, despite the simple idea of just working with a subset

322 of files, there are a range of different behavioral changes that need to be

@@ -326,37 +329,38 @@ understanding these differences can be beneficial.

329

330 * Commands behaving the same regardless of high-level use-case

331

329 - * commands that only look at files within the sparsity specification

332 + ** commands that only look at files within the sparsity specification

333

331 - * diff (without --cached or REVISION arguments)

332 - * grep (without --cached or REVISION arguments)

333 - * diff-files

334 + *** diff (without --cached or REVISION arguments)

335 + *** grep (without --cached or REVISION arguments)

336 + *** diff-files

337

335 - * commands that restore files to the working tree that match sparsity

338 + ** commands that restore files to the working tree that match sparsity

339 patterns, and remove unmodified files that don't match those

340 patterns:

341

339 - * switch

340 - * checkout (the switch-like half)

341 - * read-tree

342 - * reset --hard

342 + *** switch

343 + *** checkout (the switch-like half)

344 + *** read-tree

345 + *** reset --hard

346

344 - * commands that write conflicted files to the working tree, but otherwise

347 + ** commands that write conflicted files to the working tree, but otherwise

348 will omit writing files to the working tree that do not match the

349 sparsity patterns:

350

348 - * merge

349 - * rebase

350 - * cherry-pick

351 - * revert

351 + *** merge

352 + *** rebase

353 + *** cherry-pick

354 + *** revert

355

353 - * `am` and `apply --cached` should probably be in this section but

356 + *** `am` and `apply --cached` should probably be in this section but

357 are buggy (see the "Known bugs" section below)

358

359 The behavior for these commands somewhat depends upon the merge

360 strategy being used:

358 - * `ort` behaves as described above

359 - * `octopus` and `resolve` will always vivify any file changed in the merge

361 +

362 + *** `ort` behaves as described above

363 + *** `octopus` and `resolve` will always vivify any file changed in the merge

364 relative to the first parent, which is rather suboptimal.

365

366 It is also important to note that these commands WILL update the index

@@ -372,21 +376,21 @@ understanding these differences can be beneficial.

376 specification and the sparsity patterns (much like the commands in the

377 previous section).

378

375 - * commands that always ignore sparsity since commits must be full-tree

379 + ** commands that always ignore sparsity since commits must be full-tree

380

377 - * archive

378 - * bundle

379 - * commit

380 - * format-patch

381 - * fast-export

382 - * fast-import

383 - * commit-tree

381 + *** archive

382 + *** bundle

383 + *** commit

384 + *** format-patch

385 + *** fast-export

386 + *** fast-import

387 + *** commit-tree

388

385 - * commands that write any modified file to the working tree (conflicted

389 + ** commands that write any modified file to the working tree (conflicted

390 or not, and whether those paths match sparsity patterns or not):

391

388 - * stash

389 - * apply (without `--index` or `--cached`)

392 + *** stash

393 + *** apply (without `--index` or `--cached`)

394

395 * Commands that may slightly differ for behavior A vs. behavior B:

396

@@ -394,19 +398,20 @@ understanding these differences can be beneficial.

398 behaviors, but may differ in verbosity and types of warning and error

399 messages.

400

397 - * commands that make modifications to which files are tracked:

398 - * add

399 - * rm

400 - * mv

401 - * update-index

401 + ** commands that make modifications to which files are tracked:

402 +

403 + *** add

404 + *** rm

405 + *** mv

406 + *** update-index

407

408 The fact that files can move between the 'tracked' and 'untracked'

409 categories means some commands will have to treat untracked files

410 differently. But if we have to treat untracked files differently,

411 then additional commands may also need changes:

412

408 - * status

409 - * clean

413 + *** status

414 + *** clean

415

416 In particular, `status` may need to report any untracked files outside

417 the sparsity specification as an erroneous condition (especially to

@@ -420,9 +425,10 @@ understanding these differences can be beneficial.

425 may need to ignore the sparse specification by its nature. Also, its

426 current --[no-]ignore-skip-worktree-entries default is totally bogus.

427

423 - * commands for manually tweaking paths in both the index and the working tree

424 - * `restore`

425 - * the restore-like half of `checkout`

428 + ** commands for manually tweaking paths in both the index and the working tree

429 +

430 + *** `restore`

431 + *** the restore-like half of `checkout`

432

433 These commands should be similar to add/rm/mv in that they should

434 only operate on the sparse specification by default, and require a

@@ -433,18 +439,19 @@ understanding these differences can be beneficial.

439

440 * Commands that significantly differ for behavior A vs. behavior B:

441

436 - * commands that query history

437 - * diff (with --cached or REVISION arguments)

438 - * grep (with --cached or REVISION arguments)

439 - * show (when given commit arguments)

440 - * blame (only matters when one or more -C flags are passed)

441 - * and annotate

442 - * log

443 - * whatchanged (may not exist anymore)

444 - * ls-files

445 - * diff-index

446 - * diff-tree

447 - * ls-tree

442 + ** commands that query history

443 +

444 + *** diff (with --cached or REVISION arguments)

445 + *** grep (with --cached or REVISION arguments)

446 + *** show (when given commit arguments)

447 + *** blame (only matters when one or more -C flags are passed)

448 + **** and annotate

449 + *** log

450 + *** whatchanged (may not exist anymore)

451 + *** ls-files

452 + *** diff-index

453 + *** diff-tree

454 + *** ls-tree

455

456 Note: for log and whatchanged, revision walking logic is unaffected

457 but displaying of patches is affected by scoping the command to the

@@ -458,91 +465,91 @@ understanding these differences can be beneficial.

465

466 * Commands I don't know how to classify

467

461 - * range-diff

468 + ** range-diff

469

470 Is this like `log` or `format-patch`?

471

465 - * cherry

472 + ** cherry

473

474 See range-diff

475

476 * Commands unaffected by sparse-checkouts

477

471 - * shortlog

472 - * show-branch

473 - * rev-list

474 - * bisect

475 -

476 - * branch

477 - * describe

478 - * fetch

479 - * gc

480 - * init

481 - * maintenance

482 - * notes

483 - * pull (merge & rebase have the necessary changes)

484 - * push

485 - * submodule

486 - * tag

487 -

488 - * config

489 - * filter-branch (works in separate checkout without sparse-checkout setup)

490 - * pack-refs

491 - * prune

492 - * remote

493 - * repack

494 - * replace

495 -

496 - * bugreport

497 - * count-objects

498 - * fsck

499 - * gitweb

500 - * help

501 - * instaweb

502 - * merge-tree (doesn't touch worktree or index, and merges always compute full-tree)

503 - * rerere

504 - * verify-commit

505 - * verify-tag

506 -

507 - * commit-graph

508 - * hash-object

509 - * index-pack

510 - * mktag

511 - * mktree

512 - * multi-pack-index

513 - * pack-objects

514 - * prune-packed

515 - * symbolic-ref

516 - * unpack-objects

517 - * update-ref

518 - * write-tree (operates on index, possibly optimized to use sparse dir entries)

519 -

520 - * for-each-ref

521 - * get-tar-commit-id

522 - * ls-remote

523 - * merge-base (merges are computed full tree, so merge base should be too)

524 - * name-rev

525 - * pack-redundant

526 - * rev-parse

527 - * show-index

528 - * show-ref

529 - * unpack-file

530 - * var

531 - * verify-pack

532 -

533 - * <Everything under 'Interacting with Others' in 'git help --all'>

534 - * <Everything under 'Low-level...Syncing' in 'git help --all'>

535 - * <Everything under 'Low-level...Internal Helpers' in 'git help --all'>

536 - * <Everything under 'External commands' in 'git help --all'>

478 + ** shortlog

479 + ** show-branch

480 + ** rev-list

481 + ** bisect

482 +

483 + ** branch

484 + ** describe

485 + ** fetch

486 + ** gc

487 + ** init

488 + ** maintenance

489 + ** notes

490 + ** pull (merge & rebase have the necessary changes)

491 + ** push

492 + ** submodule

493 + ** tag

494 +

495 + ** config

496 + ** filter-branch (works in separate checkout without sparse-checkout setup)

497 + ** pack-refs

498 + ** prune

499 + ** remote

500 + ** repack

501 + ** replace

502 +

503 + ** bugreport

504 + ** count-objects

505 + ** fsck

506 + ** gitweb

507 + ** help

508 + ** instaweb

509 + ** merge-tree (doesn't touch worktree or index, and merges always compute full-tree)

510 + ** rerere

511 + ** verify-commit

512 + ** verify-tag

513 +

514 + ** commit-graph

515 + ** hash-object

516 + ** index-pack

517 + ** mktag

518 + ** mktree

519 + ** multi-pack-index

520 + ** pack-objects

521 + ** prune-packed

522 + ** symbolic-ref

523 + ** unpack-objects

524 + ** update-ref

525 + ** write-tree (operates on index, possibly optimized to use sparse dir entries)

526 +

527 + ** for-each-ref

528 + ** get-tar-commit-id

529 + ** ls-remote

530 + ** merge-base (merges are computed full tree, so merge base should be too)

531 + ** name-rev

532 + ** pack-redundant

533 + ** rev-parse

534 + ** show-index

535 + ** show-ref

536 + ** unpack-file

537 + ** var

538 + ** verify-pack

539 +

540 + ** <Everything under 'Interacting with Others' in 'git help --all'>

541 + ** <Everything under 'Low-level...Syncing' in 'git help --all'>

542 + ** <Everything under 'Low-level...Internal Helpers' in 'git help --all'>

543 + ** <Everything under 'External commands' in 'git help --all'>

544

545 * Commands that might be affected, but who cares?

546

540 - * merge-file

541 - * merge-index

542 - * gitk?

547 + ** merge-file

548 + ** merge-index

549 + ** gitk?

550

551

545 -=== Behavior classes ===

552 +== Behavior classes ==

553

554 From the above there are a few classes of behavior:

555

@@ -573,18 +580,19 @@ From the above there are a few classes of behavior:

580

581 Commands in this class generally behave like the "restrict" class,

582 except that:

576 - (1) they will ignore the sparse specification and write files with

577 - conflicts to the working tree (thus temporarily expanding the

578 - sparse specification to include such files.)

579 - (2) they are grouped with commands which move to a new commit, since

580 - they often create a commit and then move to it, even though we

581 - know there are many exceptions to moving to the new commit. (For

582 - example, the user may rebase a commit that becomes empty, or have

583 - a cherry-pick which conflicts, or a user could run `merge

584 - --no-commit`, and we also view `apply --index` kind of like `am

585 - --no-commit`.) As such, these commands can make changes to index

586 - files outside the sparse specification, though they'll mark such

587 - files with SKIP_WORKTREE.

583 +

584 + (1) they will ignore the sparse specification and write files with

585 + conflicts to the working tree (thus temporarily expanding the

586 + sparse specification to include such files.)

587 + (2) they are grouped with commands which move to a new commit, since

588 + they often create a commit and then move to it, even though we

589 + know there are many exceptions to moving to the new commit. (For

590 + example, the user may rebase a commit that becomes empty, or have

591 + a cherry-pick which conflicts, or a user could run `merge

592 + --no-commit`, and we also view `apply --index` kind of like `am

593 + --no-commit`.) As such, these commands can make changes to index

594 + files outside the sparse specification, though they'll mark such

595 + files with SKIP_WORKTREE.

596

597 * "restrict also specially applied to untracked files"

598

@@ -609,37 +617,39 @@ From the above there are a few classes of behavior:

617 specification.

618

619

612 -=== Subcommand-dependent defaults ===

620 +== Subcommand-dependent defaults ==

621

622 Note that we have different defaults depending on the command for the

623 desired behavior :

624

625 * Commands defaulting to "restrict":

618 - * diff-files

619 - * diff (without --cached or REVISION arguments)

620 - * grep (without --cached or REVISION arguments)

621 - * switch

622 - * checkout (the switch-like half)

623 - * reset (<commit>)

624 -

625 - * restore

626 - * checkout (the restore-like half)

627 - * checkout-index

628 - * reset (with pathspec)

626 +

627 + ** diff-files

628 + ** diff (without --cached or REVISION arguments)

629 + ** grep (without --cached or REVISION arguments)

630 + ** switch

631 + ** checkout (the switch-like half)

632 + ** reset (<commit>)

633 +

634 + ** restore

635 + ** checkout (the restore-like half)

636 + ** checkout-index

637 + ** reset (with pathspec)

638

639 This behavior makes sense; these interact with the working tree.

640

641 * Commands defaulting to "restrict modulo conflicts":

633 - * merge

634 - * rebase

635 - * cherry-pick

636 - * revert

642

638 - * am

639 - * apply --index (which is kind of like an `am --no-commit`)

643 + ** merge

644 + ** rebase

645 + ** cherry-pick

646 + ** revert

647 +

648 + ** am

649 + ** apply --index (which is kind of like an `am --no-commit`)

650

641 - * read-tree (especially with -m or -u; is kind of like a --no-commit merge)

642 - * reset (<tree-ish>, due to similarity to read-tree)

651 + ** read-tree (especially with -m or -u; is kind of like a --no-commit merge)

652 + ** reset (<tree-ish>, due to similarity to read-tree)

653

654 These also interact with the working tree, but require slightly

655 different behavior either so that (a) conflicts can be resolved or (b)

@@ -648,16 +658,17 @@ desired behavior :

658 (See also the "Known bugs" section below regarding `am` and `apply`)

659

660 * Commands defaulting to "no restrict":

651 - * archive

652 - * bundle

653 - * commit

654 - * format-patch

655 - * fast-export

656 - * fast-import

657 - * commit-tree

661

659 - * stash

660 - * apply (without `--index`)

662 + ** archive

663 + ** bundle

664 + ** commit

665 + ** format-patch

666 + ** fast-export

667 + ** fast-import

668 + ** commit-tree

669 +

670 + ** stash

671 + ** apply (without `--index`)

672

673 These have completely different defaults and perhaps deserve the most

674 detailed explanation:

@@ -679,53 +690,59 @@ desired behavior :

690 sparse specification then we'll lose changes from the user.

691

692 * Commands defaulting to "restrict also specially applied to untracked files":

682 - * add

683 - * rm

684 - * mv

685 - * update-index

686 - * status

687 - * clean (?)

688 -

689 - Our original implementation for the first three of these commands was

690 - "no restrict", but it had some severe usability issues:

691 - * `git add <somefile>` if honored and outside the sparse

692 - specification, can result in the file randomly disappearing later

693 - when some subsequent command is run (since various commands

694 - automatically clean up unmodified files outside the sparse

695 - specification).

696 - * `git rm '*.jpg'` could very negatively surprise users if it deletes

697 - files outside the range of the user's interest.

698 - * `git mv` has similar surprises when moving into or out of the cone,

699 - so best to restrict by default

700 -

701 - So, we switched `add` and `rm` to default to "restrict", which made

702 - usability problems much less severe and less frequent, but we still got

703 - complaints because commands like:

704 - git add <file-outside-sparse-specification>

705 - git rm <file-outside-sparse-specification>

706 - would silently do nothing. We should instead print an error in those

707 - cases to get usability right.

708 -

709 - update-index needs to be updated to match, and status and maybe clean

710 - also need to be updated to specially handle untracked paths.

711 -

712 - There may be a difference in here between behavior A and behavior B in

713 - terms of verboseness of errors or additional warnings.

693 +

694 + ** add

695 + ** rm

696 + ** mv

697 + ** update-index

698 + ** status

699 + ** clean (?)

700 +

701 +....

702 + Our original implementation for the first three of these commands was

703 + "no restrict", but it had some severe usability issues:

704 +

705 + * `git add <somefile>` if honored and outside the sparse

706 + specification, can result in the file randomly disappearing later

707 + when some subsequent command is run (since various commands

708 + automatically clean up unmodified files outside the sparse

709 + specification).

710 + * `git rm '*.jpg'` could very negatively surprise users if it deletes

711 + files outside the range of the user's interest.

712 + * `git mv` has similar surprises when moving into or out of the cone,

713 + so best to restrict by default

714 +

715 + So, we switched `add` and `rm` to default to "restrict", which made

716 + usability problems much less severe and less frequent, but we still got

717 + complaints because commands like:

718 +

719 + git add <file-outside-sparse-specification>

720 + git rm <file-outside-sparse-specification>

721 +

722 + would silently do nothing. We should instead print an error in those

723 + cases to get usability right.

724 +

725 + update-index needs to be updated to match, and status and maybe clean

726 + also need to be updated to specially handle untracked paths.

727 +

728 + There may be a difference in here between behavior A and behavior B in

729 + terms of verboseness of errors or additional warnings.

730 +....

731

732 * Commands falling under "restrict or no restrict dependent upon behavior

733 A vs. behavior B"

734

718 - * diff (with --cached or REVISION arguments)

719 - * grep (with --cached or REVISION arguments)

720 - * show (when given commit arguments)

721 - * blame (only matters when one or more -C flags passed)

722 - * and annotate

723 - * log

724 - * and variants: shortlog, gitk, show-branch, whatchanged, rev-list

725 - * ls-files

726 - * diff-index

727 - * diff-tree

728 - * ls-tree

735 + ** diff (with --cached or REVISION arguments)

736 + ** grep (with --cached or REVISION arguments)

737 + ** show (when given commit arguments)

738 + ** blame (only matters when one or more -C flags passed)

739 + *** and annotate

740 + ** log

741 + *** and variants: shortlog, gitk, show-branch, whatchanged, rev-list

742 + ** ls-files

743 + ** diff-index

744 + ** diff-tree

745 + ** ls-tree

746

747 For now, we default to behavior B for these, which want a default of

748 "no restrict".

@@ -749,7 +766,7 @@ desired behavior :

766 implemented.

767

768

752 -=== Sparse specification vs. sparsity patterns ===

769 +== Sparse specification vs. sparsity patterns ==

770

771 In a well-behaved situation, the sparse specification is given directly

772 by the $GIT_DIR/info/sparse-checkout file. However, it can transiently

@@ -821,45 +838,48 @@ under behavior B index operations are lumped with history and tend to

838 operate full-tree.

839

840

824 -=== Implementation Questions ===

825 -

826 - * Do the options --scope={sparse,all} sound good to others? Are there better

827 - options?

828 - * Names in use, or appearing in patches, or previously suggested:

829 - * --sparse/--dense

830 - * --ignore-skip-worktree-bits

831 - * --ignore-skip-worktree-entries

832 - * --ignore-sparsity

833 - * --[no-]restrict-to-sparse-paths

834 - * --full-tree/--sparse-tree

835 - * --[no-]restrict

836 - * --scope={sparse,all}

837 - * --focus/--unfocus

838 - * --limit/--unlimited

839 - * Rationale making me lean slightly towards --scope={sparse,all}:

840 - * We want a name that works for many commands, so we need a name that

841 +== Implementation Questions ==

842 +

843 + * Do the options --scope={sparse,all} sound good to others? Are there better options?

844 +

845 + ** Names in use, or appearing in patches, or previously suggested:

846 +

847 + *** --sparse/--dense

848 + *** --ignore-skip-worktree-bits

849 + *** --ignore-skip-worktree-entries

850 + *** --ignore-sparsity

851 + *** --[no-]restrict-to-sparse-paths

852 + *** --full-tree/--sparse-tree

853 + *** --[no-]restrict

854 + *** --scope={sparse,all}

855 + *** --focus/--unfocus

856 + *** --limit/--unlimited

857 +

858 + ** Rationale making me lean slightly towards --scope={sparse,all}:

859 +

860 + *** We want a name that works for many commands, so we need a name that

861 does not conflict

842 - * We know that we have more than two possible usecases, so it is best

862 + *** We know that we have more than two possible usecases, so it is best

863 to avoid a flag that appears to be binary.

844 - * --scope={sparse,all} isn't overly long and seems relatively

864 + *** --scope={sparse,all} isn't overly long and seems relatively

865 explanatory

846 - * `--sparse`, as used in add/rm/mv, is totally backwards for

866 + *** `--sparse`, as used in add/rm/mv, is totally backwards for

867 grep/log/etc. Changing the meaning of `--sparse` for these

868 commands would fix the backwardness, but possibly break existing

869 scripts. Using a new name pairing would allow us to treat

870 `--sparse` in these commands as a deprecated alias.

851 - * There is a different `--sparse`/`--dense` pair for commands using

871 + *** There is a different `--sparse`/`--dense` pair for commands using

872 revision machinery, so using that naming might cause confusion

853 - * There is also a `--sparse` in both pack-objects and show-branch, which

873 + *** There is also a `--sparse` in both pack-objects and show-branch, which

874 don't conflict but do suggest that `--sparse` is overloaded

855 - * The name --ignore-skip-worktree-bits is a double negative, is

875 + *** The name --ignore-skip-worktree-bits is a double negative, is

876 quite a mouthful, refers to an implementation detail that many

877 users may not be familiar with, and we'd need a negation for it

878 which would probably be even more ridiculously long. (But we

879 can make --ignore-skip-worktree-bits a deprecated alias for

880 --no-restrict.)

881

862 - * If a config option is added (sparse.scope?) what should the values and

882 + ** If a config option is added (sparse.scope?) what should the values and

883 description be? "sparse" (behavior A), "worktree-sparse-history-dense"

884 (behavior B), "dense" (behavior C)? There's a risk of confusion,

885 because even for Behaviors A and B we want some commands to be

@@ -868,19 +888,20 @@ operate full-tree.

888 the primary difference we are focusing is just the history-querying

889 commands (log/diff/grep). Previous config suggestion here: [13]

890

871 - * Is `--no-expand` a good alias for ls-files's `--sparse` option?

891 + ** Is `--no-expand` a good alias for ls-files's `--sparse` option?

892 (`--sparse` does not map to either `--scope=sparse` or `--scope=all`,

893 because in non-cone mode it does nothing and in cone-mode it shows the

894 sparse directory entries which are technically outside the sparse

895 specification)

896

877 - * Under Behavior A:

878 - * Does ls-files' `--no-expand` override the default `--scope=all`, or

897 + ** Under Behavior A:

898 +

899 + *** Does ls-files' `--no-expand` override the default `--scope=all`, or

900 does it need an extra flag?

880 - * Does ls-files' `-t` option imply `--scope=all`?

881 - * Does update-index's `--[no-]skip-worktree` option imply `--scope=all`?

901 + *** Does ls-files' `-t` option imply `--scope=all`?

902 + *** Does update-index's `--[no-]skip-worktree` option imply `--scope=all`?

903

883 - * sparse-checkout: once behavior A is fully implemented, should we take

904 + ** sparse-checkout: once behavior A is fully implemented, should we take

905 an interim measure to ease people into switching the default? Namely,

906 if folks are not already in a sparse checkout, then require

907 `sparse-checkout init/set` to take a

@@ -892,7 +913,7 @@ operate full-tree.

913 is seamless for them.

914

915

895 -=== Implementation Goals/Plans ===

916 +== Implementation Goals/Plans ==

917

918 * Get buy-in on this document in general.

919

@@ -910,25 +931,26 @@ operate full-tree.

931 request that they not trigger this bug." flag

932

933 * Flags & Config

913 - * Make `--sparse` in add/rm/mv a deprecated alias for `--scope=all`

914 - * Make `--ignore-skip-worktree-bits` in checkout-index/checkout/restore

934 +

935 + ** Make `--sparse` in add/rm/mv a deprecated alias for `--scope=all`

936 + ** Make `--ignore-skip-worktree-bits` in checkout-index/checkout/restore

937 a deprecated aliases for `--scope=all`

916 - * Create config option (sparse.scope?), tie it to the "Cliff notes"

938 + ** Create config option (sparse.scope?), tie it to the "Cliff notes"

939 overview

940

919 - * Add --scope=sparse (and --scope=all) flag to each of the history querying

941 + ** Add --scope=sparse (and --scope=all) flag to each of the history querying

942 commands. IMPORTANT: make sure diff machinery changes don't mess with

943 format-patch, fast-export, etc.

944

923 -=== Known bugs ===

945 +== Known bugs ==

946

947 This list used to be a lot longer (see e.g. [1,2,3,4,5,6,7,8,9]), but we've

948 been working on it.

949

928 -0. Behavior A is not well supported in Git. (Behavior B didn't used to

950 +1. Behavior A is not well supported in Git. (Behavior B didn't used to

951 be either, but was the easier of the two to implement.)

952

931 -1. am and apply:

953 +2. am and apply:

954

955 apply, without `--index` or `--cached`, relies on files being present

956 in the working copy, and also writes to them unconditionally. As

@@ -948,7 +970,7 @@ been working on it.

970 files and then complain that those vivified files would be

971 overwritten by merge.

972

951 -2. reset --hard:

973 +3. reset --hard:

974

975 reset --hard provides confusing error message (works correctly, but

976 misleads the user into believing it didn't):

@@ -971,13 +993,13 @@ been working on it.

993 `git reset --hard` DID remove addme from the index and the working tree, contrary

994 to the error message, but in line with how reset --hard should behave.

995

974 -3. read-tree

996 +4. read-tree

997

998 `read-tree` doesn't apply the 'SKIP_WORKTREE' bit to *any* of the

999 entries it reads into the index, resulting in all your files suddenly

1000 appearing to be "deleted".

1001

980 -4. Checkout, restore:

1002 +5. Checkout, restore:

1003

1004 These command do not handle path & revision arguments appropriately:

1005

@@ -1030,7 +1052,7 @@ been working on it.

1052 S tracked

1053 H tracked-but-maybe-skipped

1054

1033 -5. checkout and restore --staged, continued:

1055 +6. checkout and restore --staged, continued:

1056

1057 These commands do not correctly scope operations to the sparse

1058 specification, and make it worse by not setting important SKIP_WORKTREE

@@ -1046,56 +1068,82 @@ been working on it.

1068 the sparse specification, but then it will be important to set the

1069 SKIP_WORKTREE bits appropriately.

1070

1049 -6. Performance issues; see:

1050 - https://lore.kernel.org/git/CABPp-BEkJQoKZsQGCYioyga_uoDQ6iBeW+FKr8JhyuuTMK1RDw@mail.gmail.com/

1071 +7. Performance issues; see:

1072 +

1073 + https://lore.kernel.org/git/CABPp-BEkJQoKZsQGCYioyga_uoDQ6iBeW+FKr8JhyuuTMK1RDw@mail.gmail.com/

1074

1075

1053 -=== Reference Emails ===

1076 +== Reference Emails ==

1077

1078 Emails that detail various bugs we've had in sparse-checkout:

1079

1057 -[1] (Original descriptions of behavior A & behavior B)

1058 - https://lore.kernel.org/git/CABPp-BGJ_Nvi5TmgriD9Bh6eNXE2EDq2f8e8QKXAeYG3BxZafA@mail.gmail.com/

1059 -[2] (Fix stash applications in sparse checkouts; bugs from behavioral differences)

1060 - https://lore.kernel.org/git/ccfedc7140dbf63ba26a15f93bd3885180b26517.1606861519.git.gitgitgadget@gmail.com/

1061 -[3] (Present-despite-skipped entries)

1062 - https://lore.kernel.org/git/11d46a399d26c913787b704d2b7169cafc28d639.1642175983.git.gitgitgadget@gmail.com/

1063 -[4] (Clone --no-checkout interaction)

1064 - https://lore.kernel.org/git/pull.801.v2.git.git.1591324899170.gitgitgadget@gmail.com/ (clone --no-checkout)

1065 -[5] (The need for update_sparsity() and avoiding `read-tree -mu HEAD`)

1066 - https://lore.kernel.org/git/3a1f084641eb47515b5a41ed4409a36128913309.1585270142.git.gitgitgadget@gmail.com/

1067 -[6] (SKIP_WORKTREE is advisory, not mandatory)

1068 - https://lore.kernel.org/git/844306c3e86ef67591cc086decb2b760e7d710a3.1585270142.git.gitgitgadget@gmail.com/

1069 -[7] (`worktree add` should copy sparsity settings from current worktree)

1070 - https://lore.kernel.org/git/c51cb3714e7b1d2f8c9370fe87eca9984ff4859f.1644269584.git.gitgitgadget@gmail.com/

1071 -[8] (Avoid negative surprises in add, rm, and mv)

1072 - https://lore.kernel.org/git/cover.1617914011.git.matheus.bernardino@usp.br/

1073 - https://lore.kernel.org/git/pull.1018.v4.git.1632497954.gitgitgadget@gmail.com/

1074 -[9] (Move from out-of-cone to in-cone)

1075 - https://lore.kernel.org/git/20220630023737.473690-6-shaoxuan.yuan02@gmail.com/

1076 - https://lore.kernel.org/git/20220630023737.473690-4-shaoxuan.yuan02@gmail.com/

1077 -[10] (Unnecessarily downloading objects outside sparse specification)

1078 - https://lore.kernel.org/git/CAOLTT8QfwOi9yx_qZZgyGa8iL8kHWutEED7ok_jxwTcYT_hf9Q@mail.gmail.com/

1079 -

1080 -[11] (Stolee's comments on high-level usecases)

1081 - https://lore.kernel.org/git/1a1e33f6-3514-9afc-0a28-5a6b85bd8014@gmail.com/

1080 +[1] (Original descriptions of behavior A & behavior B):

1081 +

1082 +https://lore.kernel.org/git/CABPp-BGJ_Nvi5TmgriD9Bh6eNXE2EDq2f8e8QKXAeYG3BxZafA@mail.gmail.com/

1083 +

1084 +[2] (Fix stash applications in sparse checkouts; bugs from behavioral differences):

1085 +

1086 +https://lore.kernel.org/git/ccfedc7140dbf63ba26a15f93bd3885180b26517.1606861519.git.gitgitgadget@gmail.com/

1087 +

1088 +[3] (Present-despite-skipped entries):

1089 +

1090 +https://lore.kernel.org/git/11d46a399d26c913787b704d2b7169cafc28d639.1642175983.git.gitgitgadget@gmail.com/

1091 +

1092 +[4] (Clone --no-checkout interaction):

1093 +

1094 +https://lore.kernel.org/git/pull.801.v2.git.git.1591324899170.gitgitgadget@gmail.com/ (clone --no-checkout)

1095 +

1096 +[5] (The need for update_sparsity() and avoiding `read-tree -mu HEAD`):

1097 +

1098 +https://lore.kernel.org/git/3a1f084641eb47515b5a41ed4409a36128913309.1585270142.git.gitgitgadget@gmail.com/

1099 +

1100 +[6] (SKIP_WORKTREE is advisory, not mandatory):

1101 +

1102 +https://lore.kernel.org/git/844306c3e86ef67591cc086decb2b760e7d710a3.1585270142.git.gitgitgadget@gmail.com/

1103 +

1104 +[7] (`worktree add` should copy sparsity settings from current worktree):

1105 +

1106 +https://lore.kernel.org/git/c51cb3714e7b1d2f8c9370fe87eca9984ff4859f.1644269584.git.gitgitgadget@gmail.com/

1107 +

1108 +[8] (Avoid negative surprises in add, rm, and mv):

1109 +

1110 + * https://lore.kernel.org/git/cover.1617914011.git.matheus.bernardino@usp.br/

1111 + * https://lore.kernel.org/git/pull.1018.v4.git.1632497954.gitgitgadget@gmail.com/

1112 +

1113 +[9] (Move from out-of-cone to in-cone):

1114 +

1115 + * https://lore.kernel.org/git/20220630023737.473690-6-shaoxuan.yuan02@gmail.com/

1116 + * https://lore.kernel.org/git/20220630023737.473690-4-shaoxuan.yuan02@gmail.com/

1117 +

1118 +[10] (Unnecessarily downloading objects outside sparse specification):

1119 +

1120 +https://lore.kernel.org/git/CAOLTT8QfwOi9yx_qZZgyGa8iL8kHWutEED7ok_jxwTcYT_hf9Q@mail.gmail.com/

1121 +

1122 +[11] (Stolee's comments on high-level usecases):

1123 +

1124 +https://lore.kernel.org/git/1a1e33f6-3514-9afc-0a28-5a6b85bd8014@gmail.com/

1125

1126 [12] Others commenting on eventually switching default to behavior A:

1127 +

1128 * https://lore.kernel.org/git/xmqqh719pcoo.fsf@gitster.g/

1129 * https://lore.kernel.org/git/xmqqzgeqw0sy.fsf@gitster.g/

1130 * https://lore.kernel.org/git/a86af661-cf58-a4e5-0214-a67d3a794d7e@github.com/

1131

1088 -[13] Previous config name suggestion and description

1089 - * https://lore.kernel.org/git/CABPp-BE6zW0nJSStcVU=_DoDBnPgLqOR8pkTXK3dW11=T01OhA@mail.gmail.com/

1132 +[13] Previous config name suggestion and description:

1133 +

1134 + https://lore.kernel.org/git/CABPp-BE6zW0nJSStcVU=_DoDBnPgLqOR8pkTXK3dW11=T01OhA@mail.gmail.com/

1135

1136 [14] Tangential issue: switch to cone mode as default sparse specification mechanism:

1092 - https://lore.kernel.org/git/a1b68fd6126eb341ef3637bb93fedad4309b36d0.1650594746.git.gitgitgadget@gmail.com/

1137 +

1138 +https://lore.kernel.org/git/a1b68fd6126eb341ef3637bb93fedad4309b36d0.1650594746.git.gitgitgadget@gmail.com/

1139

1140 [15] Lengthy email on grep behavior, covering what should be searched:

1095 - * https://lore.kernel.org/git/CABPp-BGVO3QdbfE84uF_3QDF0-y2iHHh6G5FAFzNRfeRitkuHw@mail.gmail.com/

1141 +

1142 +https://lore.kernel.org/git/CABPp-BGVO3QdbfE84uF_3QDF0-y2iHHh6G5FAFzNRfeRitkuHw@mail.gmail.com/

1143

1144 [16] Email explaining sparsity patterns vs. SKIP_WORKTREE and history operations,

1145 search for the parenthetical comment starting "We do not check".

1099 - https://lore.kernel.org/git/CABPp-BFsCPPNOZ92JQRJeGyNd0e-TCW-LcLyr0i_+VSQJP+GCg@mail.gmail.com/

1146 +

1147 +https://lore.kernel.org/git/CABPp-BFsCPPNOZ92JQRJeGyNd0e-TCW-LcLyr0i_+VSQJP+GCg@mail.gmail.com/

1148

1149 [17] https://lore.kernel.org/git/20220207190320.2960362-1-jonathantanmy@google.com/

doc: sparse-checkout.adoc: fix asciidoc warnings