@matheusbsbs / matheus-bagliano / commits / 725ca8a8e7
Fork 0

Add Subproject Design Notes.

Signed-off-by: Junio C Hamano <junkio@cox.net>

Junio C Hamano committed Jan 22, 2006 at 23:53 UTC 725ca8a8e77847049630b1f409c28fb37e943dc2
2 files changed +483
Makefile new
+11
@@ -0,0 +1,11 @@
1 +all:
2 +
3 +clean:
4 + rm -f Subpro.html
5 +
6 +
7 +all: Subpro.html
8 +
9 +%.html: %.txt
10 + asciidoc -bxhtml11 $*.txt
11 +
Subpro.txt new
+472
@@ -0,0 +1,472 @@
1 +Notes on Subproject Support
2 +===========================
3 +Junio C Hamano
4 +
5 +Scenario
6 +--------
7 +
8 +The examples in the following discussion show how this proposal
9 +plans to help this:
10 +
11 +. A project to build an embedded Linux appliance "gadget" is
12 + maintained with git.
13 +
14 +. The project uses linux-2.6 kernel as its subcomponent. It
15 + starts from a particular version of the mainline kernel, but
16 + adds its own code and build infrastructure to fit the
17 + appliance's needs.
18 +
19 +. The working tree of the project is laid out this way:
20 ++
21 +------------
22 + Makefile - Builds the whole thing.
23 + linux-2.6/ - The kernel, perhaps modified for the project.
24 + appliance/ - Applications that run on the appliance, and
25 + other bits.
26 +------------
27 +
28 +. The project is willing to maintain its own changes out of tree
29 + of the Linux kernel project, but would want to be able to feed
30 + the changes upstream, and incorporate upstream changes to its
31 + own tree, taking advantage of the fact that both itself and
32 + the Linux kernel project are version controlled with git.
33 +
34 +. To make the story a bit more interesting, later in the history
35 + of development, `linux-2.6/` and `appliance/` directories will
36 + be renamed to `kernel/` and `gadget/`.
37 +
38 +The idea here is to:
39 +
40 +. Keep `linux-2.6/` part as an independent project. The work by
41 + the project on the kernel part can be naturally exchanged with
42 + the other kernel developers this way. Specifically, a tree
43 + object contained in commit objects belonging to this project
44 + does *not* have `linux-2.6/` directory at the top.
45 +
46 +. Keep the `appliance/` part as another independent project.
47 + Applications are supposed to be more or less independent from
48 + the kernel version, but some other bits might be tied to a
49 + specific kernel version. Again, a tree object contained in
50 + commit objects belonging to this project does *not* have
51 + `appliance/` directory at the top.
52 +
53 +. Have another project that combines the whole thing together,
54 + so that the project can keep track of which versions of the
55 + parts are built together.
56 +
57 +We will call the project that binds things together the
58 +'toplevel project'. Other projects that hold `linux-2.6/` part
59 +and `appliance/` part are called 'subprojects'.
60 +
61 +
62 +Setting up
63 +----------
64 +
65 +Let's say we have been working on the appliance software,
66 +independently version controlled with git. Also the kernel part
67 +has been version controlled separately, like this:
68 +------------
69 +$ ls -dF current/*/.git current/*
70 +current/Makefile current/appliance/.git/ current/linux-2.6/.git/
71 +current/appliance/ current/linux-2.6/
72 +------------
73 +
74 +Now we would want to get a combined project. First we would
75 +clone from these repositories (which is not strictly needed --
76 +we could use `$GIT_ALTERNATE_OBJECT_DIRECTORIES` instead):
77 +
78 +------------
79 +$ mkdir combined && cd combined
80 +$ cp ../current/Makefile .
81 +$ git init-db
82 +$ mkdir -p .git/refs/subs/{kernel,gadget}/{heads,tags}
83 +$ git clone-pack ../current/linux-2.6/ master | read kernel_commit junk
84 +$ git clone-pack ../current/appliance/ master | read gadget_commit junk
85 +------------
86 +
87 +We will introduce a new command to set up a combined project:
88 +
89 +------------
90 +$ git bind-projects \
91 + $kernel_commit linux-2.6/ \
92 + $gadget_commit appliance/
93 +------------
94 +
95 +This would probably do an equivalent of:
96 +
97 +------------
98 +$ rm -f "$GIT_DIR/index"
99 +$ git read-tree --prefix=linux-2.6/ $kernel_commit
100 +$ git read-tree --prefix=appliance/ $gadget_commit
101 +$ git update-index --bind linux-2.6/ $kernel_commit
102 +$ git update-index --bind appliance/ $gadget_commit
103 +------------
104 +[NOTE]
105 +============
106 +Earlier outlines sent to the git mailing list talked
107 +about `$GIT_DIR/bind` to record what subproject are bound to
108 +which subtree in the current working tree and index. This
109 +proposal instead records that information in the index file
110 +with `update-index --bind` command.
111 +
112 +Also note that in this round of proposal, there is no separate
113 +branches that keep track of heads of subprojects.
114 +============
115 +
116 +Let's not forget to add the `Makefile`, and check the whole
117 +thing out from the index file.
118 +------------
119 +$ git add Makefile
120 +$ git checkout-index -f -u -q -a
121 +------------
122 +
123 +Now our directory should be identical with the `current`
124 +directory. After making sure of that, we should be able to
125 +commit the whole thing:
126 +
127 +------------
128 +$ diff -x .git -r ../current ../combined
129 +$ git commit -m 'Initial toplevel project commit'
130 +------------
131 +
132 +Which should create a new commit object that records what is in
133 +the index file as its tree, with `bind` lines to record which
134 +subproject commit objects are bound at what subdirectory, and
135 +updates the `$GIT_DIR/refs/heads/master`. Such a commit object
136 +might look like this:
137 +------------
138 +tree 04803b09c300c8325258ccf2744115acc4c57067
139 +bind 5b2bcc7b2d546c636f79490655b3347acc91d17f linux-2.6/
140 +bind 0bdd79af62e8621359af08f0afca0ce977348ac7 appliance/
141 +author Junio C Hamano <junio@kernel.org> 1137965565 -0800
142 +committer Junio C Hamano <junio@kernel.org> 1137965565 -0800
143 +
144 +Initial toplevel project commit
145 +------------
146 +
147 +Notice that `Makefile` at the top is part of the toplevel
148 +project in this example, but it is not necessary. We could
149 +instead have the appliance subproject include this file. In
150 +such a setup, the appliance subproject would have had `Makefile`
151 +and `appliance/` directory at the toplevel. The `bind` line for
152 +that project would have said "the rest is bound at `/`" and
153 +`write-tree \--exclude=linux-2.6/` would have been used to write
154 +the tree for that subproject out of the combined index.
155 +
156 +
157 +Making further commits
158 +----------------------
159 +
160 +The easiest case is when you updated the Makefile without
161 +changing anything in the subprojects. In such a case, we just
162 +need to create a new commmit object that records the new tree
163 +with the current `HEAD` as its parent, and with the same set of
164 +`bind` lines.
165 +
166 +When we have changes to the subproject part, we would make a
167 +separate commit to the subproject part and then record the whole
168 +thing by making a commit to the toplevel project. The user
169 +interaction might go this way:
170 +------------
171 +$ git commit
172 +error: you have changes to the subproject bound at linux-2.6/.
173 +$ git commit --subproject linux-2.6/
174 +$ git commit
175 +------------
176 +
177 +With the new `\--subproject` option, the directory structure
178 +rooted at `linux-2.6/` part is written out as a tree, and a new
179 +commit object that records that tree object with the commit
180 +bound to that portion of the tree (`5b2bcc7b` in the above
181 +example) as its parent is created. Then the final `git commit`
182 +would record the whole tree with updated `bind` line for the
183 +`linux-2.6/` part.
184 +
185 +
186 +Checking out
187 +------------
188 +
189 +After cloning such a toplevel project, `git clone` without `-n`
190 +option would check out the working tree. This is done by
191 +reading the tree object recorded in the commit object (which
192 +records the whole thing), and adding the information from the
193 +"bind" line to the index file.
194 +
195 +------------
196 +$ cd ..
197 +$ git clone -n combined cloned ;# clone the one we created earlier
198 +$ cd cloned
199 +$ git checkout
200 +------------
201 +
202 +This round of proposal does not maintain separate branch heads
203 +for subprojects. The bound commits and their subdirectories
204 +are recorded in the index file from the commit object, so there
205 +is no need to do anything other than updating the index and the
206 +working tree.
207 +
208 +
209 +Switching branches
210 +------------------
211 +
212 +Along with the traditional two-way merge by `read-tree -m -u`,
213 +we would need to look at:
214 +
215 +. `bind` lines in the current `HEAD` commit.
216 +
217 +. `bind` lines in the commit we are switching to.
218 +
219 +. subproject binding information in the index file.
220 +
221 +to make sure we do sensible things.
222 +
223 +Just like until very recently we did not allow switching
224 +branches when two-way merge would lose local changes, we can
225 +start by refusing to switch branches when the subprojects bound
226 +in the index do not match what is recorded in the `HEAD` commit.
227 +
228 +Because in this round of the proposal we do not use the
229 +`$GIT_DIR/bind` file nor separate branches to keep track of
230 +heads of the subprojects, there is nothing else other than the
231 +working tree and the index file that needs to be updated when
232 +switching branches.
233 +
234 +
235 +Merging
236 +-------
237 +
238 +Merging two branches of the toplevel projects can use the
239 +traditional merging mechanism mostly unchanged. The merge base
240 +computation can be done using the `parent` ancestry information
241 +taken from the two toplevel project branch heads being merged,
242 +and merging of the whole tree can be done with a three-way merge
243 +of the whole tree using the merge base and two head commits.
244 +For reasons described later, we would not merge the subproject
245 +parts of the trees during this step, though.
246 +
247 +When the two branch heads use different versions of subproject,
248 +things get a bit tricky. First, let's forget for a moment about
249 +the case where they bind the same project at different location.
250 +We would refuse if they do not have the same number of `bind`
251 +lines that bind something at the same subdirectories.
252 +
253 +------------
254 +$ git merge 'Merge in a side branch' HEAD side
255 +error: the merged heads have subprojects bound at different places.
256 + ours:
257 + linux-2.6/
258 + appliance/
259 + theirs:
260 + kernel/
261 + gadget/
262 + manual/
263 +------------
264 +
265 +Such renaming can be handled by first moving the bind points in
266 +our branch, and redoing the merge (this is a rare operation
267 +anyway). It might go like this:
268 +
269 +------------
270 +$ git reset
271 +$ git update-index --unbind linux-2.6/
272 +$ git update-index --unbind appliance/
273 +$ git update-index --bind $kernel_commit kernel/
274 +$ git update-index --bind $gadget_commit gadget/
275 +$ git commit -m 'Prepare for merge with side branch'
276 +$ git merge 'Merge in a side branch' HEAD side
277 +error: the merged heads have subprojects bound at different places.
278 + ours:
279 + kernel/
280 + gadget/
281 + theirs:
282 + kernel/
283 + gadget/
284 + manual/
285 +------------
286 +
287 +Their branch added another subproject, so this did not work (or
288 +it could be the other way around -- we might have been the one
289 +with `manual/` subproject while they didn't). This suggests
290 +that we may want an option to `git merge` to allow taking a
291 +union of subprojects. Again, this is a rare operation, and
292 +always taking a union would have created a toplevel project that
293 +had both `kernel/` and `linux-2.6/` bound to the same Linux
294 +kernel project from possibly different vintage, so it would be
295 +prudent to require the set of bound subprojects to exactly match
296 +and give the user an option to take a union.
297 +
298 +------------
299 +$ git merge --union-subprojects 'Merge in a side branch HEAD side
300 +error: the subproject at 'kernel/' needs to be merged first.
301 +------------
302 +
303 +Here, the version of the Linux kernel project in the `side`
304 +branch was different from what our branch had on our `bind`
305 +line. On what kind of difference should we give this error?
306 +Initially, I think we could require one is the fast forward of
307 +the other (ours might be ahead of theirs, or the other way
308 +around), and take the descendant.
309 +
310 +Or we could do an independent merge of subprojects heads, using
311 +the `parent` ancestry of the bound subproject heads to find
312 +their merge-base and doing a three-way merge. This would leave
313 +the merge result in the subproject part of the working tree and
314 +the index.
315 +
316 +[NOTE]
317 +This is the reason we did not do the whole-tree three way merge
318 +earlier. The subproject commit bound to the merge base commit
319 +used for the toplevel project may not be the merge base between
320 +the subproject commits bound to the two toplevel project
321 +commits.
322 +
323 +So let's deal with the case to merge only a subproject part into
324 +our tree first.
325 +
326 +
327 +Merging subprojects
328 +-------------------
329 +
330 +An operation of more practical importance is to be able to merge
331 +in changes done outside to the projects bound to our toplevel
332 +project.
333 +
334 +------------
335 +$ git pull --subproject=kernel/ git://git.kernel.org/.../linux-2.6/
336 +------------
337 +
338 +might do:
339 +
340 +. fetch the current `HEAD` commit from Linus.
341 +. find the subproject commit bound at kernel/ subtree.
342 +. perform the usual three-way merge of these two commits, in
343 + `kernel/` part of the working tree.
344 +
345 +After that, `git commit \--subproject` option would be needed to
346 +make a commit.
347 +
348 +[NOTE]
349 +This suggests that we would need to have something similar to
350 +`MERGE_HEAD` for merging the subproject part. In the case of
351 +merging two toplevel project commits, we probably can read the
352 +`bind` lines from the `MERGE_HEAD` commit and either our `HEAD`
353 +commit or our index file. Further, we probably would require
354 +that the latter two must match, just as we currently require the
355 +index file matches our `HEAD` commit before `git merge`.
356 +
357 +Just like the current `pull = fetch + merge` semantics, the
358 +subproject aware version `git pull \--subproject=frotz/` would be
359 +a `git fetch \--subproject=frotz/` followed by a `git merge
360 +\--subproject=frotz/`. So the above would be:
361 +
362 +. Fetch the head.
363 ++
364 +------------
365 +$ git fetch --subproject=kernel/ git://git.kernel.org/.../linux-2.6/
366 +------------
367 ++
368 +which would fetch the commit chain from the remote repository, and
369 +write something like this to `FETCH_HEAD`:
370 ++
371 +------------
372 +3ee68c4...\tfor-merge-into kernel/\tbranch 'master' of git://.../linux-2.6
373 +------------
374 +
375 +. Run `git merge`.
376 ++
377 +------------
378 +$ git merge --subproject=kernel/ \
379 + 'Merge git://.../linux-2.6 into kernel/' HEAD 3ee68c4...
380 +------------
381 +
382 +. In case it does not cleanly automerge, `git merge` would write
383 +the necessary information for a later `git commit` to use in
384 +`MERGE_HEAD`. It may look like this:
385 ++
386 +------------
387 +3ee68c4af3fd7228c1be63254b9f884614f9ebb2 kernel/
388 +------------
389 ++
390 +Similarly, `MERGE_MSG` file will hold the merge message.
391 +
392 +With this, a later invocation of `git commit` to record the
393 +result of hand resolving would be able to notice that:
394 +
395 +. We should be first resolving `kernel/` subproject, not the
396 + whole thing.
397 +. The remote `HEAD` is `3ee68c4\...` commit.
398 +. The merge message is `Merge git://\.../linux-2.6 into kernel/`.
399 +
400 +and would make a merge commit, and register that resulting
401 +commit in the index file using `update-index \--bind` instead of
402 +updating *any* branch head.
403 +
404 +
405 +Management of Subprojects
406 +-------------------------
407 +
408 +While the above as a mechanism would support version controlling
409 +of subprojects as a part of *one* larger toplevel project, it
410 +probably is worth pointing out that having a separate repository
411 +to manage the subproject independently would be a good idea.
412 +The same subproject can be incorporated into more than one
413 +toplevel projects, and after all, a subproject should be
414 +something that can stand on its own. In our example scenario,
415 +the `kernel/` project is used as a subproject for the "gadget"
416 +product, but at the same time, the organizaton that runs the
417 +"gadget" project may use Linux on their development machines,
418 +and have their own kernel hackers, not necessarily related to
419 +the use of the kernel in the "gadget" product.
420 +
421 +What this suggests is that not just we need to be able to pull
422 +the kernel development history *into* the subproject of the
423 +"gadget" project, but also we need to be able to push the
424 +development history of the kernel part alone *out* *of* the
425 +"gadget" project to another repository that deals only with the
426 +kernel part.
427 +
428 +It might go this way. First the setup:
429 +
430 +------------
431 +$ git clone git://git.kernel.org/.../linux-2.6 Linux
432 +$ ls -dF *
433 +cloned/ combined/ current/ Linux/
434 +------------
435 +
436 +That is, in addition to the `combined/` which we have been using
437 +to develop the "gadget" product in, we now have a repository for
438 +the kernel, cloned from Linus. In the previous section, we have
439 +outlined how we update the kernel subproject part of `combined/`
440 +repository from the `kernel.org` repository. The same procedure
441 +would work for pulling from `Linux/` repository here.
442 +
443 +We are now going the other way; propagate the kernel work done
444 +in the "gadget" project repository `combined/` back to `Linux/`.
445 +We might do this at the lowest level:
446 +
447 +------------
448 +$ cd combined
449 +$ git cat-file commit HEAD |
450 + sed -ne 's|^bind \([0-9a-f]*\) kernel/$|\1|p' >.git/refs/heads/linux26
451 +$ git push ../Linux linux26:master
452 +------------
453 +
454 +Or, more realistically, since the `Linux` project might already
455 +have their own commits on its `master`:
456 +
457 +------------
458 +$ cd Linux
459 +$ git pull ../combined linux26
460 +------------
461 +
462 +Either way we would need an easy way to maintain the `linux26`
463 +branch in the above example, and that will have to be part of
464 +the wrapper scripts like `git commit` (more likely, that would
465 +be a job for `git commit \--subproject`) for the usability's
466 +sake; in other words, the `cat-file commit` piped to `sed` above
467 +is not something the end user would do, but something that is
468 +done by the wrapper scripts.
469 +
470 +Hopefully the people who work in `Linux/` repository would run
471 +`format-patch` and feed their changes back to the kernel
472 +community.