commit-graph: merge commit-graph chains
When searching for a commit in a commit-graph chain of G graphs with N commits, the search takes O(G log N) time. If we always add a new tip graph with every write, the linear G term will start to dominate and slow the lookup process. To keep lookups fast, but also keep most incremental writes fast, create a strategy for merging levels of the commit-graph chain. The strategy is detailed in the commit-graph design document, but is summarized by these two conditions: 1. If the number of commits we are adding is more than half the number of commits in the graph below, then merge with that graph. 2. If we are writing more than 64,000 commits into a single graph, then merge with all lower graphs. The numeric values in the conditions above are currently constant, but can become config options in a future update. As we merge levels of the commit-graph chain, check that the commits still exist in the repository. A garbage-collection operation may have removed those commits from the object store and we do not want to persist them in the commit-graph chain. This is a non-issue if the 'git gc' process wrote a new, single-level commit-graph file. After we merge levels, the old graph-{hash}.graph files are no longer referenced by the commit-graph-chain file. We will expire these files in a future change. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Derrick Stolee committed
Jun 18, 2019 at 11:14 UTC
1771be90c8e4797c2466296d1d570dbfa39d9743
3 files changed
+240
-33
Documentation/technical/commit-graph.txt
+80
@@ -186,6 +186,86 @@ positions to refer to their parents, which may be in `graph-{hash1}.graph` or
186
its containment in the intervals [0, X0), [X0, X0 + X1), [X0 + X1, X0 + X1 +
187
X2).
188
189
+Each commit-graph file (except the base, `graph-{hash0}.graph`) contains data
190
+specifying the hashes of all files in the lower layers. In the above example,
191
+`graph-{hash1}.graph` contains `{hash0}` while `graph-{hash2}.graph` contains
192
+`{hash0}` and `{hash1}`.
193
+
194
+## Merging commit-graph files
195
+
196
+If we only added a new commit-graph file on every write, we would run into a
197
+linear search problem through many commit-graph files. Instead, we use a merge
198
+strategy to decide when the stack should collapse some number of levels.
199
+
200
+The diagram below shows such a collapse. As a set of new commits are added, it
201
+is determined by the merge strategy that the files should collapse to
202
+`graph-{hash1}`. Thus, the new commits, the commits in `graph-{hash2}` and
203
+the commits in `graph-{hash1}` should be combined into a new `graph-{hash3}`
204
+file.
205
+
206
+ +---------------------+
207
+ | |
208
+ | (new commits) |
209
+ | |
210
+ +---------------------+
211
+ | |
212
+ +-----------------------+ +---------------------+
213
+ | graph-{hash2} |->| |
214
+ +-----------------------+ +---------------------+
215
+ | | |
216
+ +-----------------------+ +---------------------+
217
+ | | | |
218
+ | graph-{hash1} |->| |
219
+ | | | |
220
+ +-----------------------+ +---------------------+
221
+ | tmp_graphXXX
222
+ +-----------------------+
223
+ | |
224
+ | |
225
+ | |
226
+ | graph-{hash0} |
227
+ | |
228
+ | |
229
+ | |
230
+ +-----------------------+
231
+
232
+During this process, the commits to write are combined, sorted and we write the
233
+contents to a temporary file, all while holding a `commit-graph-chain.lock`
234
+lock-file. When the file is flushed, we rename it to `graph-{hash3}`
235
+according to the computed `{hash3}`. Finally, we write the new chain data to
236
+`commit-graph-chain.lock`:
237
+
238
+```
239
+ {hash3}
240
+ {hash0}
241
+```
242
+
243
+We then close the lock-file.
244
+
245
+## Merge Strategy
246
+
247
+When writing a set of commits that do not exist in the commit-graph stack of
248
+height N, we default to creating a new file at level N + 1. We then decide to
249
+merge with the Nth level if one of two conditions hold:
250
+
251
+ 1. The expected file size for level N + 1 is at least half the file size for
252
+ level N.
253
+
254
+ 2. Level N + 1 contains more than 64,0000 commits.
255
+
256
+This decision cascades down the levels: when we merge a level we create a new
257
+set of commits that then compares to the next level.
258
+
259
+The first condition bounds the number of levels to be logarithmic in the total
260
+number of commits. The second condition bounds the total number of commits in
261
+a `graph-{hashN}` file and not in the `commit-graph` file, preventing
262
+significant performance issues when the stack merges and another process only
263
+partially reads the previous stack.
264
+
265
+The merge strategy values (2 for the size multiple, 64,000 for the maximum
266
+number of commits) could be extracted into config settings for full
267
+flexibility.
268
+
269
Related Links
270
-------------
271
[0] https://bugs.chromium.org/p/git/issues/detail?id=8
commit-graph.c
+147
-33
@@ -1298,36 +1298,6 @@ static int write_graph_chunk_base(struct hashfile *f,
1298
return 0;
1299
}
1300
1301
-static void init_commit_graph_chain(struct write_commit_graph_context *ctx)
1302
-{
1303
- struct commit_graph *g = ctx->r->objects->commit_graph;
1304
- uint32_t i;
1305
-
1306
- ctx->new_base_graph = g;
1307
- ctx->base_graph_name = xstrdup(g->filename);
1308
- ctx->new_num_commits_in_base = g->num_commits + g->num_commits_in_base;
1309
-
1310
- ctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;
1311
-
1312
- ALLOC_ARRAY(ctx->commit_graph_filenames_after, ctx->num_commit_graphs_after);
1313
- ALLOC_ARRAY(ctx->commit_graph_hash_after, ctx->num_commit_graphs_after);
1314
-
1315
- for (i = 0; i < ctx->num_commit_graphs_before - 1; i++)
1316
- ctx->commit_graph_filenames_after[i] = xstrdup(ctx->commit_graph_filenames_before[i]);
1317
-
1318
- if (ctx->num_commit_graphs_before)
1319
- ctx->commit_graph_filenames_after[ctx->num_commit_graphs_before - 1] =
1320
- get_split_graph_filename(ctx->obj_dir, oid_to_hex(&g->oid));
1321
-
1322
- i = ctx->num_commit_graphs_before - 1;
1323
-
1324
- while (g) {
1325
- ctx->commit_graph_hash_after[i] = xstrdup(oid_to_hex(&g->oid));
1326
- i--;
1327
- g = g->base_graph;
1328
- }
1329
-}
1330
-
1301
static int write_commit_graph_file(struct write_commit_graph_context *ctx)
1302
{
1303
uint32_t i;
@@ -1509,6 +1479,145 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)
1479
return 0;
1480
}
1481
1482
+static int split_strategy_max_commits = 64000;
1483
+static float split_strategy_size_mult = 2.0f;
1484
+
1485
+static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)
1486
+{
1487
+ struct commit_graph *g = ctx->r->objects->commit_graph;
1488
+ uint32_t num_commits = ctx->commits.nr;
1489
+ uint32_t i;
1490
+
1491
+ g = ctx->r->objects->commit_graph;
1492
+ ctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;
1493
+
1494
+ while (g && (g->num_commits <= split_strategy_size_mult * num_commits ||
1495
+ num_commits > split_strategy_max_commits)) {
1496
+ num_commits += g->num_commits;
1497
+ g = g->base_graph;
1498
+
1499
+ ctx->num_commit_graphs_after--;
1500
+ }
1501
+
1502
+ ctx->new_base_graph = g;
1503
+
1504
+ ALLOC_ARRAY(ctx->commit_graph_filenames_after, ctx->num_commit_graphs_after);
1505
+ ALLOC_ARRAY(ctx->commit_graph_hash_after, ctx->num_commit_graphs_after);
1506
+
1507
+ for (i = 0; i < ctx->num_commit_graphs_after &&
1508
+ i < ctx->num_commit_graphs_before; i++)
1509
+ ctx->commit_graph_filenames_after[i] = xstrdup(ctx->commit_graph_filenames_before[i]);
1510
+
1511
+ i = ctx->num_commit_graphs_before - 1;
1512
+ g = ctx->r->objects->commit_graph;
1513
+
1514
+ while (g) {
1515
+ if (i < ctx->num_commit_graphs_after)
1516
+ ctx->commit_graph_hash_after[i] = xstrdup(oid_to_hex(&g->oid));
1517
+
1518
+ i--;
1519
+ g = g->base_graph;
1520
+ }
1521
+}
1522
+
1523
+static void merge_commit_graph(struct write_commit_graph_context *ctx,
1524
+ struct commit_graph *g)
1525
+{
1526
+ uint32_t i;
1527
+ uint32_t offset = g->num_commits_in_base;
1528
+
1529
+ ALLOC_GROW(ctx->commits.list, ctx->commits.nr + g->num_commits, ctx->commits.alloc);
1530
+
1531
+ for (i = 0; i < g->num_commits; i++) {
1532
+ struct object_id oid;
1533
+ struct commit *result;
1534
+
1535
+ display_progress(ctx->progress, i + 1);
1536
+
1537
+ load_oid_from_graph(g, i + offset, &oid);
1538
+
1539
+ /* only add commits if they still exist in the repo */
1540
+ result = lookup_commit_reference_gently(ctx->r, &oid, 1);
1541
+
1542
+ if (result) {
1543
+ ctx->commits.list[ctx->commits.nr] = result;
1544
+ ctx->commits.nr++;
1545
+ }
1546
+ }
1547
+}
1548
+
1549
+static int commit_compare(const void *_a, const void *_b)
1550
+{
1551
+ const struct commit *a = *(const struct commit **)_a;
1552
+ const struct commit *b = *(const struct commit **)_b;
1553
+ return oidcmp(&a->object.oid, &b->object.oid);
1554
+}
1555
+
1556
+static void sort_and_scan_merged_commits(struct write_commit_graph_context *ctx)
1557
+{
1558
+ uint32_t i, num_parents;
1559
+ struct commit_list *parent;
1560
+
1561
+ if (ctx->report_progress)
1562
+ ctx->progress = start_delayed_progress(
1563
+ _("Scanning merged commits"),
1564
+ ctx->commits.nr);
1565
+
1566
+ QSORT(ctx->commits.list, ctx->commits.nr, commit_compare);
1567
+
1568
+ ctx->num_extra_edges = 0;
1569
+ for (i = 0; i < ctx->commits.nr; i++) {
1570
+ display_progress(ctx->progress, i);
1571
+
1572
+ if (i && oideq(&ctx->commits.list[i - 1]->object.oid,
1573
+ &ctx->commits.list[i]->object.oid)) {
1574
+ die(_("unexpected duplicate commit id %s"),
1575
+ oid_to_hex(&ctx->commits.list[i]->object.oid));
1576
+ } else {
1577
+ num_parents = 0;
1578
+ for (parent = ctx->commits.list[i]->parents; parent; parent = parent->next)
1579
+ num_parents++;
1580
+
1581
+ if (num_parents > 2)
1582
+ ctx->num_extra_edges += num_parents - 2;
1583
+ }
1584
+ }
1585
+
1586
+ stop_progress(&ctx->progress);
1587
+}
1588
+
1589
+static void merge_commit_graphs(struct write_commit_graph_context *ctx)
1590
+{
1591
+ struct commit_graph *g = ctx->r->objects->commit_graph;
1592
+ uint32_t current_graph_number = ctx->num_commit_graphs_before;
1593
+ struct strbuf progress_title = STRBUF_INIT;
1594
+
1595
+ while (g && current_graph_number >= ctx->num_commit_graphs_after) {
1596
+ current_graph_number--;
1597
+
1598
+ if (ctx->report_progress) {
1599
+ strbuf_addstr(&progress_title, _("Merging commit-graph"));
1600
+ ctx->progress = start_delayed_progress(progress_title.buf, 0);
1601
+ }
1602
+
1603
+ merge_commit_graph(ctx, g);
1604
+ stop_progress(&ctx->progress);
1605
+ strbuf_release(&progress_title);
1606
+
1607
+ g = g->base_graph;
1608
+ }
1609
+
1610
+ if (g) {
1611
+ ctx->new_base_graph = g;
1612
+ ctx->new_num_commits_in_base = g->num_commits + g->num_commits_in_base;
1613
+ }
1614
+
1615
+ if (ctx->new_base_graph)
1616
+ ctx->base_graph_name = xstrdup(ctx->new_base_graph->filename);
1617
+
1618
+ sort_and_scan_merged_commits(ctx);
1619
+}
1620
+
1621
int write_commit_graph(const char *obj_dir,
1622
struct string_list *pack_indexes,
1623
struct string_list *commit_hex,
@@ -1554,6 +1663,9 @@ int write_commit_graph(const char *obj_dir,
1663
ctx->approx_nr_objects = approximate_object_count();
1664
ctx->oids.alloc = ctx->approx_nr_objects / 32;
1665
1666
+ if (ctx->split && ctx->oids.alloc > split_strategy_max_commits)
1667
+ ctx->oids.alloc = split_strategy_max_commits;
1668
+
1669
if (ctx->append) {
1670
prepare_commit_graph_one(ctx->r, ctx->obj_dir);
1671
if (ctx->r->objects->commit_graph)
@@ -1607,9 +1719,11 @@ int write_commit_graph(const char *obj_dir,
1719
if (!ctx->commits.nr)
1720
goto cleanup;
1721
1610
- if (ctx->split)
1611
- init_commit_graph_chain(ctx);
1612
- else
1722
+ if (ctx->split) {
1723
+ split_graph_merge_strategy(ctx);
1724
+
1725
+ merge_commit_graphs(ctx);
1726
+ } else
1727
ctx->num_commit_graphs_after = 1;
1728
1729
compute_generation_numbers(ctx);
t/t5324-split-commit-graph.sh
+13
@@ -119,4 +119,17 @@ test_expect_success 'add one commit, write a tip graph' '
119
120
graph_git_behavior 'three-layer commit-graph: commit 11 vs 6' commits/11 commits/6
121
122
+test_expect_success 'add one commit, write a merged graph' '
123
+ test_commit 12 &&
124
+ git branch commits/12 &&
125
+ git commit-graph write --reachable --split &&
126
+ test_path_is_file $graphdir/commit-graph-chain &&
127
+ test_line_count = 2 $graphdir/commit-graph-chain &&
128
+ ls $graphdir/graph-*.graph >graph-files &&
129
+ test_line_count = 4 graph-files &&
130
+ verify_chain_files_exist $graphdir
131
+'
132
+
133
+graph_git_behavior 'merged commit-graph: commit 12 vs 6' commits/12 commits/6
134
+
135
test_done