@matheusbsbs / matheus-bagliano / commits / 20b20a22f8

upload-pack: provide a hook for running pack-objects

When upload-pack serves a client request, it turns to pack-objects to do the heavy lifting of creating a packfile. There's no easy way to intercept the call to pack-objects, but there are a few good reasons to want to do so: 1. If you're debugging a client or server issue with fetching, you may want to store a copy of the generated packfile. 2. If you're gathering data from real-world fetches for performance analysis or debugging, storing a copy of the arguments and stdin lets you replay the pack generation at your leisure. 3. You may want to insert a caching layer around pack-objects; it is the most CPU- and memory-intensive part of serving a fetch, and its output is a pure function[1] of its input, making it an ideal place to consolidate identical requests. This patch adds a simple "hook" interface to intercept calls to pack-objects. The new test demonstrates how it can be used for debugging (using it for caching is a straightforward extension; the tricky part is writing the actual caching layer). This hook is unlike the normal hook scripts found in the "hooks/" directory of a repository. Because we promise that upload-pack is safe to run in an untrusted repository, we cannot execute arbitrary code or commands found in the repository (neither in hooks/, nor in the config). So instead, this hook is triggered from a config variable that is explicitly ignored in the per-repo config. The config variable holds the actual shell command to run as the hook. Another approach would be to simply treat it as a boolean: "should I respect the upload-pack hooks in this repo?", and then run the script from "hooks/" as we usually do. However, that isn't as flexible; there's no way to run a hook approved by the site administrator (e.g., in "/etc/gitconfig") on a repository whose contents are not trusted. The approach taken by this patch is more fine-grained, if a little less conventional for git hooks (it does behave similar to other configured commands like diff.external, etc). [1] Pack-objects isn't _actually_ a pure function. Its output depends on the exact packing of the object database, and if multi-threading is used for delta compression, can even differ racily. But for the purposes of caching, that's OK; of the many possible outputs for a given input, it is sufficient only that we output one of them. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Jeff King committed May 18, 2016 at 18:45 UTC 20b20a22f8f7c1420e259c97ef790cb93091f475

3 files changed +89 -1

Documentation/config.txt

+15

@@ -2880,6 +2880,21 @@ uploadpack.keepAlive::

2880 `uploadpack.keepAlive` seconds. Setting this option to 0

2881 disables keepalive packets entirely. The default is 5 seconds.

2882

2883 +uploadpack.packObjectsHook::

2884 + If this option is set, when `upload-pack` would run

2885 + `git pack-objects` to create a packfile for a client, it will

2886 + run this shell command instead. The `pack-objects` command and

2887 + arguments it _would_ have run (including the `git pack-objects`

2888 + at the beginning) are appended to the shell command. The stdin

2889 + and stdout of the hook are treated as if `pack-objects` itself

2890 + was run. I.e., `upload-pack` will feed input intended for

2891 + `pack-objects` to the hook, and expects a completed packfile on

2892 + stdout.

2893 ++

2894 +Note that this configuration variable is ignored if it is seen in the

2895 +repository-level config (this is a safety measure against fetching from

2896 +untrusted repositories).

2897 +

2898 url.<base>.insteadOf::

2899 Any URL that starts with this value will be rewritten to

2900 start, instead, with <base>. In cases where some site serves a

t/t5544-pack-objects-hook.sh new

+62

@@ -0,0 +1,62 @@

1 +#!/bin/sh

2 +

3 +test_description='test custom script in place of pack-objects'

4 +. ./test-lib.sh

5 +

6 +test_expect_success 'create some history to fetch' '

7 + test_commit one &&

8 + test_commit two

9 +'

10 +

11 +test_expect_success 'create debugging hook script' '

12 + write_script .git/hook <<-\EOF

13 + echo >&2 "hook running"

14 + echo "$*" >hook.args

15 + cat >hook.stdin

16 + "$@" <hook.stdin >hook.stdout

17 + cat hook.stdout

18 + EOF

19 +'

20 +

21 +clear_hook_results () {

22 + rm -rf .git/hook.* dst.git

23 +}

24 +

25 +test_expect_success 'hook runs via global config' '

26 + clear_hook_results &&

27 + test_config_global uploadpack.packObjectsHook ./hook &&

28 + git clone --no-local . dst.git 2>stderr &&

29 + grep "hook running" stderr

30 +'

31 +

32 +test_expect_success 'hook outputs are sane' '

33 + # check that we recorded a usable pack

34 + git index-pack --stdin <.git/hook.stdout &&

35 +

36 + # check that we recorded args and stdin. We do not check

37 + # the full argument list or the exact pack contents, as it would make

38 + # the test brittle. So just sanity check that we could replay

39 + # the packing procedure.

40 + grep "^git" .git/hook.args &&

41 + $(cat .git/hook.args) <.git/hook.stdin >replay

42 +'

43 +

44 +test_expect_success 'hook runs from -c config' '

45 + clear_hook_results &&

46 + git clone --no-local \

47 + -u "git -c uploadpack.packObjectsHook=./hook upload-pack" \

48 + . dst.git 2>stderr &&

49 + grep "hook running" stderr

50 +'

51 +

52 +test_expect_success 'hook does not run from repo config' '

53 + clear_hook_results &&

54 + test_config uploadpack.packObjectsHook "./hook" &&

55 + git clone --no-local . dst.git 2>stderr &&

56 + ! grep "hook running" stderr &&

57 + test_path_is_missing .git/hook.args &&

58 + test_path_is_missing .git/hook.stdin &&

59 + test_path_is_missing .git/hook.stdout

60 +'

61 +

62 +test_done

upload-pack.c

+12 -1

@@ -52,6 +52,7 @@ static int keepalive = 5;

52 static int use_sideband;

53 static int advertise_refs;

54 static int stateless_rpc;

55 +static const char *pack_objects_hook;

57 static void reset_timeout(void)

58 {

@@ -93,6 +94,14 @@ static void create_pack_file(void)

94 int i;

95 FILE *pipe_fd;

97 + if (!pack_objects_hook)

98 + pack_objects.git_cmd = 1;

99 + else {

100 + argv_array_push(&pack_objects.args, pack_objects_hook);

101 + argv_array_push(&pack_objects.args, "git");

102 + pack_objects.use_shell = 1;

103 + }

104 +

105 if (shallow_nr) {

106 argv_array_push(&pack_objects.args, "--shallow-file");

107 argv_array_push(&pack_objects.args, "");

@@ -115,7 +124,6 @@ static void create_pack_file(void)

124 pack_objects.in = -1;

125 pack_objects.out = -1;

126 pack_objects.err = -1;

118 - pack_objects.git_cmd = 1;

127

128 if (start_command(&pack_objects))

129 die("git upload-pack: unable to fork git-pack-objects");

@@ -812,6 +820,9 @@ static int upload_pack_config(const char *var, const char *value, void *unused)

820 keepalive = git_config_int(var, value);

821 if (!keepalive)

822 keepalive = -1;

823 + } else if (current_config_scope() != CONFIG_SCOPE_REPO) {

824 + if (!strcmp("uploadpack.packobjectshook", var))

825 + return git_config_string(&pack_objects_hook, var, value);

826 }

827 return parse_hide_refs_config(var, value, "uploadpack");

828 }