Add Subproject Design Notes.
Signed-off-by: Junio C Hamano <junkio@cox.net>
Junio C Hamano committed
Jan 22, 2006 at 23:53 UTC
725ca8a8e77847049630b1f409c28fb37e943dc2
2 files changed
+483
Makefile
new
+11
@@ -0,0 +1,11 @@
1
+all:
2
+
3
+clean:
4
+ rm -f Subpro.html
5
+
6
+
7
+all: Subpro.html
8
+
9
+%.html: %.txt
10
+ asciidoc -bxhtml11 $*.txt
11
+
Subpro.txt
new
+472
@@ -0,0 +1,472 @@
1
+Notes on Subproject Support
2
+===========================
3
+Junio C Hamano
4
+
5
+Scenario
6
+--------
7
+
8
+The examples in the following discussion show how this proposal
9
+plans to help this:
10
+
11
+. A project to build an embedded Linux appliance "gadget" is
12
+ maintained with git.
13
+
14
+. The project uses linux-2.6 kernel as its subcomponent. It
15
+ starts from a particular version of the mainline kernel, but
16
+ adds its own code and build infrastructure to fit the
17
+ appliance's needs.
18
+
19
+. The working tree of the project is laid out this way:
20
++
21
+------------
22
+ Makefile - Builds the whole thing.
23
+ linux-2.6/ - The kernel, perhaps modified for the project.
24
+ appliance/ - Applications that run on the appliance, and
25
+ other bits.
26
+------------
27
+
28
+. The project is willing to maintain its own changes out of tree
29
+ of the Linux kernel project, but would want to be able to feed
30
+ the changes upstream, and incorporate upstream changes to its
31
+ own tree, taking advantage of the fact that both itself and
32
+ the Linux kernel project are version controlled with git.
33
+
34
+. To make the story a bit more interesting, later in the history
35
+ of development, `linux-2.6/` and `appliance/` directories will
36
+ be renamed to `kernel/` and `gadget/`.
37
+
38
+The idea here is to:
39
+
40
+. Keep `linux-2.6/` part as an independent project. The work by
41
+ the project on the kernel part can be naturally exchanged with
42
+ the other kernel developers this way. Specifically, a tree
43
+ object contained in commit objects belonging to this project
44
+ does *not* have `linux-2.6/` directory at the top.
45
+
46
+. Keep the `appliance/` part as another independent project.
47
+ Applications are supposed to be more or less independent from
48
+ the kernel version, but some other bits might be tied to a
49
+ specific kernel version. Again, a tree object contained in
50
+ commit objects belonging to this project does *not* have
51
+ `appliance/` directory at the top.
52
+
53
+. Have another project that combines the whole thing together,
54
+ so that the project can keep track of which versions of the
55
+ parts are built together.
56
+
57
+We will call the project that binds things together the
58
+'toplevel project'. Other projects that hold `linux-2.6/` part
59
+and `appliance/` part are called 'subprojects'.
60
+
61
+
62
+Setting up
63
+----------
64
+
65
+Let's say we have been working on the appliance software,
66
+independently version controlled with git. Also the kernel part
67
+has been version controlled separately, like this:
68
+------------
69
+$ ls -dF current/*/.git current/*
70
+current/Makefile current/appliance/.git/ current/linux-2.6/.git/
71
+current/appliance/ current/linux-2.6/
72
+------------
73
+
74
+Now we would want to get a combined project. First we would
75
+clone from these repositories (which is not strictly needed --
76
+we could use `$GIT_ALTERNATE_OBJECT_DIRECTORIES` instead):
77
+
78
+------------
79
+$ mkdir combined && cd combined
80
+$ cp ../current/Makefile .
81
+$ git init-db
82
+$ mkdir -p .git/refs/subs/{kernel,gadget}/{heads,tags}
83
+$ git clone-pack ../current/linux-2.6/ master | read kernel_commit junk
84
+$ git clone-pack ../current/appliance/ master | read gadget_commit junk
85
+------------
86
+
87
+We will introduce a new command to set up a combined project:
88
+
89
+------------
90
+$ git bind-projects \
91
+ $kernel_commit linux-2.6/ \
92
+ $gadget_commit appliance/
93
+------------
94
+
95
+This would probably do an equivalent of:
96
+
97
+------------
98
+$ rm -f "$GIT_DIR/index"
99
+$ git read-tree --prefix=linux-2.6/ $kernel_commit
100
+$ git read-tree --prefix=appliance/ $gadget_commit
101
+$ git update-index --bind linux-2.6/ $kernel_commit
102
+$ git update-index --bind appliance/ $gadget_commit
103
+------------
104
+[NOTE]
105
+============
106
+Earlier outlines sent to the git mailing list talked
107
+about `$GIT_DIR/bind` to record what subproject are bound to
108
+which subtree in the current working tree and index. This
109
+proposal instead records that information in the index file
110
+with `update-index --bind` command.
111
+
112
+Also note that in this round of proposal, there is no separate
113
+branches that keep track of heads of subprojects.
114
+============
115
+
116
+Let's not forget to add the `Makefile`, and check the whole
117
+thing out from the index file.
118
+------------
119
+$ git add Makefile
120
+$ git checkout-index -f -u -q -a
121
+------------
122
+
123
+Now our directory should be identical with the `current`
124
+directory. After making sure of that, we should be able to
125
+commit the whole thing:
126
+
127
+------------
128
+$ diff -x .git -r ../current ../combined
129
+$ git commit -m 'Initial toplevel project commit'
130
+------------
131
+
132
+Which should create a new commit object that records what is in
133
+the index file as its tree, with `bind` lines to record which
134
+subproject commit objects are bound at what subdirectory, and
135
+updates the `$GIT_DIR/refs/heads/master`. Such a commit object
136
+might look like this:
137
+------------
138
+tree 04803b09c300c8325258ccf2744115acc4c57067
139
+bind 5b2bcc7b2d546c636f79490655b3347acc91d17f linux-2.6/
140
+bind 0bdd79af62e8621359af08f0afca0ce977348ac7 appliance/
141
+author Junio C Hamano <junio@kernel.org> 1137965565 -0800
142
+committer Junio C Hamano <junio@kernel.org> 1137965565 -0800
143
+
144
+Initial toplevel project commit
145
+------------
146
+
147
+Notice that `Makefile` at the top is part of the toplevel
148
+project in this example, but it is not necessary. We could
149
+instead have the appliance subproject include this file. In
150
+such a setup, the appliance subproject would have had `Makefile`
151
+and `appliance/` directory at the toplevel. The `bind` line for
152
+that project would have said "the rest is bound at `/`" and
153
+`write-tree \--exclude=linux-2.6/` would have been used to write
154
+the tree for that subproject out of the combined index.
155
+
156
+
157
+Making further commits
158
+----------------------
159
+
160
+The easiest case is when you updated the Makefile without
161
+changing anything in the subprojects. In such a case, we just
162
+need to create a new commmit object that records the new tree
163
+with the current `HEAD` as its parent, and with the same set of
164
+`bind` lines.
165
+
166
+When we have changes to the subproject part, we would make a
167
+separate commit to the subproject part and then record the whole
168
+thing by making a commit to the toplevel project. The user
169
+interaction might go this way:
170
+------------
171
+$ git commit
172
+error: you have changes to the subproject bound at linux-2.6/.
173
+$ git commit --subproject linux-2.6/
174
+$ git commit
175
+------------
176
+
177
+With the new `\--subproject` option, the directory structure
178
+rooted at `linux-2.6/` part is written out as a tree, and a new
179
+commit object that records that tree object with the commit
180
+bound to that portion of the tree (`5b2bcc7b` in the above
181
+example) as its parent is created. Then the final `git commit`
182
+would record the whole tree with updated `bind` line for the
183
+`linux-2.6/` part.
184
+
185
+
186
+Checking out
187
+------------
188
+
189
+After cloning such a toplevel project, `git clone` without `-n`
190
+option would check out the working tree. This is done by
191
+reading the tree object recorded in the commit object (which
192
+records the whole thing), and adding the information from the
193
+"bind" line to the index file.
194
+
195
+------------
196
+$ cd ..
197
+$ git clone -n combined cloned ;# clone the one we created earlier
198
+$ cd cloned
199
+$ git checkout
200
+------------
201
+
202
+This round of proposal does not maintain separate branch heads
203
+for subprojects. The bound commits and their subdirectories
204
+are recorded in the index file from the commit object, so there
205
+is no need to do anything other than updating the index and the
206
+working tree.
207
+
208
+
209
+Switching branches
210
+------------------
211
+
212
+Along with the traditional two-way merge by `read-tree -m -u`,
213
+we would need to look at:
214
+
215
+. `bind` lines in the current `HEAD` commit.
216
+
217
+. `bind` lines in the commit we are switching to.
218
+
219
+. subproject binding information in the index file.
220
+
221
+to make sure we do sensible things.
222
+
223
+Just like until very recently we did not allow switching
224
+branches when two-way merge would lose local changes, we can
225
+start by refusing to switch branches when the subprojects bound
226
+in the index do not match what is recorded in the `HEAD` commit.
227
+
228
+Because in this round of the proposal we do not use the
229
+`$GIT_DIR/bind` file nor separate branches to keep track of
230
+heads of the subprojects, there is nothing else other than the
231
+working tree and the index file that needs to be updated when
232
+switching branches.
233
+
234
+
235
+Merging
236
+-------
237
+
238
+Merging two branches of the toplevel projects can use the
239
+traditional merging mechanism mostly unchanged. The merge base
240
+computation can be done using the `parent` ancestry information
241
+taken from the two toplevel project branch heads being merged,
242
+and merging of the whole tree can be done with a three-way merge
243
+of the whole tree using the merge base and two head commits.
244
+For reasons described later, we would not merge the subproject
245
+parts of the trees during this step, though.
246
+
247
+When the two branch heads use different versions of subproject,
248
+things get a bit tricky. First, let's forget for a moment about
249
+the case where they bind the same project at different location.
250
+We would refuse if they do not have the same number of `bind`
251
+lines that bind something at the same subdirectories.
252
+
253
+------------
254
+$ git merge 'Merge in a side branch' HEAD side
255
+error: the merged heads have subprojects bound at different places.
256
+ ours:
257
+ linux-2.6/
258
+ appliance/
259
+ theirs:
260
+ kernel/
261
+ gadget/
262
+ manual/
263
+------------
264
+
265
+Such renaming can be handled by first moving the bind points in
266
+our branch, and redoing the merge (this is a rare operation
267
+anyway). It might go like this:
268
+
269
+------------
270
+$ git reset
271
+$ git update-index --unbind linux-2.6/
272
+$ git update-index --unbind appliance/
273
+$ git update-index --bind $kernel_commit kernel/
274
+$ git update-index --bind $gadget_commit gadget/
275
+$ git commit -m 'Prepare for merge with side branch'
276
+$ git merge 'Merge in a side branch' HEAD side
277
+error: the merged heads have subprojects bound at different places.
278
+ ours:
279
+ kernel/
280
+ gadget/
281
+ theirs:
282
+ kernel/
283
+ gadget/
284
+ manual/
285
+------------
286
+
287
+Their branch added another subproject, so this did not work (or
288
+it could be the other way around -- we might have been the one
289
+with `manual/` subproject while they didn't). This suggests
290
+that we may want an option to `git merge` to allow taking a
291
+union of subprojects. Again, this is a rare operation, and
292
+always taking a union would have created a toplevel project that
293
+had both `kernel/` and `linux-2.6/` bound to the same Linux
294
+kernel project from possibly different vintage, so it would be
295
+prudent to require the set of bound subprojects to exactly match
296
+and give the user an option to take a union.
297
+
298
+------------
299
+$ git merge --union-subprojects 'Merge in a side branch HEAD side
300
+error: the subproject at 'kernel/' needs to be merged first.
301
+------------
302
+
303
+Here, the version of the Linux kernel project in the `side`
304
+branch was different from what our branch had on our `bind`
305
+line. On what kind of difference should we give this error?
306
+Initially, I think we could require one is the fast forward of
307
+the other (ours might be ahead of theirs, or the other way
308
+around), and take the descendant.
309
+
310
+Or we could do an independent merge of subprojects heads, using
311
+the `parent` ancestry of the bound subproject heads to find
312
+their merge-base and doing a three-way merge. This would leave
313
+the merge result in the subproject part of the working tree and
314
+the index.
315
+
316
+[NOTE]
317
+This is the reason we did not do the whole-tree three way merge
318
+earlier. The subproject commit bound to the merge base commit
319
+used for the toplevel project may not be the merge base between
320
+the subproject commits bound to the two toplevel project
321
+commits.
322
+
323
+So let's deal with the case to merge only a subproject part into
324
+our tree first.
325
+
326
+
327
+Merging subprojects
328
+-------------------
329
+
330
+An operation of more practical importance is to be able to merge
331
+in changes done outside to the projects bound to our toplevel
332
+project.
333
+
334
+------------
335
+$ git pull --subproject=kernel/ git://git.kernel.org/.../linux-2.6/
336
+------------
337
+
338
+might do:
339
+
340
+. fetch the current `HEAD` commit from Linus.
341
+. find the subproject commit bound at kernel/ subtree.
342
+. perform the usual three-way merge of these two commits, in
343
+ `kernel/` part of the working tree.
344
+
345
+After that, `git commit \--subproject` option would be needed to
346
+make a commit.
347
+
348
+[NOTE]
349
+This suggests that we would need to have something similar to
350
+`MERGE_HEAD` for merging the subproject part. In the case of
351
+merging two toplevel project commits, we probably can read the
352
+`bind` lines from the `MERGE_HEAD` commit and either our `HEAD`
353
+commit or our index file. Further, we probably would require
354
+that the latter two must match, just as we currently require the
355
+index file matches our `HEAD` commit before `git merge`.
356
+
357
+Just like the current `pull = fetch + merge` semantics, the
358
+subproject aware version `git pull \--subproject=frotz/` would be
359
+a `git fetch \--subproject=frotz/` followed by a `git merge
360
+\--subproject=frotz/`. So the above would be:
361
+
362
+. Fetch the head.
363
++
364
+------------
365
+$ git fetch --subproject=kernel/ git://git.kernel.org/.../linux-2.6/
366
+------------
367
++
368
+which would fetch the commit chain from the remote repository, and
369
+write something like this to `FETCH_HEAD`:
370
++
371
+------------
372
+3ee68c4...\tfor-merge-into kernel/\tbranch 'master' of git://.../linux-2.6
373
+------------
374
+
375
+. Run `git merge`.
376
++
377
+------------
378
+$ git merge --subproject=kernel/ \
379
+ 'Merge git://.../linux-2.6 into kernel/' HEAD 3ee68c4...
380
+------------
381
+
382
+. In case it does not cleanly automerge, `git merge` would write
383
+the necessary information for a later `git commit` to use in
384
+`MERGE_HEAD`. It may look like this:
385
++
386
+------------
387
+3ee68c4af3fd7228c1be63254b9f884614f9ebb2 kernel/
388
+------------
389
++
390
+Similarly, `MERGE_MSG` file will hold the merge message.
391
+
392
+With this, a later invocation of `git commit` to record the
393
+result of hand resolving would be able to notice that:
394
+
395
+. We should be first resolving `kernel/` subproject, not the
396
+ whole thing.
397
+. The remote `HEAD` is `3ee68c4\...` commit.
398
+. The merge message is `Merge git://\.../linux-2.6 into kernel/`.
399
+
400
+and would make a merge commit, and register that resulting
401
+commit in the index file using `update-index \--bind` instead of
402
+updating *any* branch head.
403
+
404
+
405
+Management of Subprojects
406
+-------------------------
407
+
408
+While the above as a mechanism would support version controlling
409
+of subprojects as a part of *one* larger toplevel project, it
410
+probably is worth pointing out that having a separate repository
411
+to manage the subproject independently would be a good idea.
412
+The same subproject can be incorporated into more than one
413
+toplevel projects, and after all, a subproject should be
414
+something that can stand on its own. In our example scenario,
415
+the `kernel/` project is used as a subproject for the "gadget"
416
+product, but at the same time, the organizaton that runs the
417
+"gadget" project may use Linux on their development machines,
418
+and have their own kernel hackers, not necessarily related to
419
+the use of the kernel in the "gadget" product.
420
+
421
+What this suggests is that not just we need to be able to pull
422
+the kernel development history *into* the subproject of the
423
+"gadget" project, but also we need to be able to push the
424
+development history of the kernel part alone *out* *of* the
425
+"gadget" project to another repository that deals only with the
426
+kernel part.
427
+
428
+It might go this way. First the setup:
429
+
430
+------------
431
+$ git clone git://git.kernel.org/.../linux-2.6 Linux
432
+$ ls -dF *
433
+cloned/ combined/ current/ Linux/
434
+------------
435
+
436
+That is, in addition to the `combined/` which we have been using
437
+to develop the "gadget" product in, we now have a repository for
438
+the kernel, cloned from Linus. In the previous section, we have
439
+outlined how we update the kernel subproject part of `combined/`
440
+repository from the `kernel.org` repository. The same procedure
441
+would work for pulling from `Linux/` repository here.
442
+
443
+We are now going the other way; propagate the kernel work done
444
+in the "gadget" project repository `combined/` back to `Linux/`.
445
+We might do this at the lowest level:
446
+
447
+------------
448
+$ cd combined
449
+$ git cat-file commit HEAD |
450
+ sed -ne 's|^bind \([0-9a-f]*\) kernel/$|\1|p' >.git/refs/heads/linux26
451
+$ git push ../Linux linux26:master
452
+------------
453
+
454
+Or, more realistically, since the `Linux` project might already
455
+have their own commits on its `master`:
456
+
457
+------------
458
+$ cd Linux
459
+$ git pull ../combined linux26
460
+------------
461
+
462
+Either way we would need an easy way to maintain the `linux26`
463
+branch in the above example, and that will have to be part of
464
+the wrapper scripts like `git commit` (more likely, that would
465
+be a job for `git commit \--subproject`) for the usability's
466
+sake; in other words, the `cat-file commit` piped to `sed` above
467
+is not something the end user would do, but something that is
468
+done by the wrapper scripts.
469
+
470
+Hopefully the people who work in `Linux/` repository would run
471
+`format-patch` and feed their changes back to the kernel
472
+community.