fetch: add a --prune-tags option and fetch.pruneTags config

Add a --prune-tags option to git-fetch, along with fetch.pruneTags config option and a -P shorthand (-p is --prune). This allows for doing any of: git fetch -p -P git fetch --prune --prune-tags git fetch -p -P origin git fetch --prune --prune-tags origin Or simply: git config fetch.prune true && git config fetch.pruneTags true && git fetch Instead of the much more verbose: git fetch --prune origin 'refs/tags/*:refs/tags/*' '+refs/heads/*:refs/remotes/origin/*' Before this feature it was painful to support the use-case of pulling from a repo which is having both its branches *and* tags deleted regularly, and have our local references to reflect upstream. At work we create deployment tags in the repo for each rollout, and there's *lots* of those, so they're archived within weeks for performance reasons. Without this change it's hard to centrally configure such repos in /etc/gitconfig (on servers that are only used for working with them). You need to set fetch.prune=true globally, and then for each repo: git -C {} config --replace-all remote.origin.fetch "refs/tags/*:refs/tags/*" "^\+*refs/tags/\*:refs/tags/\*$" Now I can simply set fetch.pruneTags=true in /etc/gitconfig as well, and users running "git pull" will automatically get the pruning semantics I want. Even though "git remote" has corresponding "prune" and "update --prune" subcommands I'm intentionally not adding a corresponding prune-tags or "update --prune --prune-tags" mode to that command. It's advertised (as noted in my recent "git remote doc: correct dangerous lies about what prune does") as only modifying remote tracking references, whereas any --prune-tags option is always going to modify what from the user's perspective is a local copy of the tag, since there's no such thing as a remote tracking tag. Ideally add_prune_tags_to_fetch_refspec() would be something that would use ALLOC_GROW() to grow the 'fetch` member of the 'remote' struct. Instead I'm realloc-ing remote->fetch and adding the tag_refspec to the end. The reason is that parse_{fetch,push}_refspec which allocate the refspec (ultimately remote->fetch) struct are called many places that don't have access to a 'remote' struct. It would be hard to change all their callsites to be amenable to carry around the bookkeeping variables required for dynamic allocation. All the other callers of the API first incrementally construct the string version of the refspec in remote->fetch_refspec via add_fetch_refspec(), before finally calling parse_fetch_refspec() via some variation of remote_get(). It's less of a pain to deal with the one special case that needs to modify already constructed refspecs than to chase down and change all the other callsites. The API I'm adding is intentionally not generalized because if we add more of these we'd probably want to re-visit how this is done. See my "Re: [BUG] git remote prune removes local tags, depending on fetch config" (87po6ahx87.fsf@evledraar.gmail.com; https://public-inbox.org/git/87po6ahx87.fsf@evledraar.gmail.com/) for more background info. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Ævar Arnfjörð Bjarmason committed Feb 9, 2018 at 20:32 UTC 97716d217c1ea00adfc64e4f6bb85c1236d661ff
8 files changed +191 -5
Documentation/config.txt
+14
@@ -1401,6 +1401,14 @@ fetch.prune::
1401 option was given on the command line. See also `remote.<name>.prune`
1402 and the PRUNING section of linkgit:git-fetch[1].
1403
1404 +fetch.pruneTags::
1405 + If true, fetch will automatically behave as if the
1406 + `refs/tags/*:refs/tags/*` refspec was provided when pruning,
1407 + if not set already. This allows for setting both this option
1408 + and `fetch.prune` to maintain a 1=1 mapping to upstream
1409 + refs. See also `remote.<name>.pruneTags` and the PRUNING
1410 + section of linkgit:git-fetch[1].
1411 +
1412 fetch.output::
1413 Control how ref update status is printed. Valid values are
1414 `full` and `compact`. Default value is `full`. See section
@@ -2945,6 +2953,12 @@ remote.<name>.prune::
2953 remove any remote-tracking references that no longer exist on the
2954 remote (as if the `--prune` option was given on the command line).
2955 Overrides `fetch.prune` settings, if any.
2956 +
2957 +remote.<name>.pruneTags::
2958 + When set to true, fetching from this remote by default will also
2959 + remove any local tags that no longer exist on the remote if pruning
2960 + is activated in general via `remote.<name>.prune`, `fetch.prune` or
2961 + `--prune`. Overrides `fetch.pruneTags` settings, if any.
2962 +
2963 See also `remote.<name>.prune` and the PRUNING section of
2964 linkgit:git-fetch[1].
Documentation/fetch-options.txt
+13 -1
@@ -73,7 +73,19 @@ ifndef::git-pull[]
73 are fetched due to an explicit refspec (either on the command
74 line or in the remote configuration, for example if the remote
75 was cloned with the --mirror option), then they are also
76 - subject to pruning.
76 + subject to pruning. Supplying `--prune-tags` is a shorthand for
77 + providing the tag refspec.
78 ++
79 +See the PRUNING section below for more details.
80 +
81 +-P::
82 +--prune-tags::
83 + Before fetching, remove any local tags that no longer exist on
84 + the remote if `--prune` is enabled. This option should be used
85 + more carefully, unlike `--prune` it will remove any local
86 + references (local tags) that have been created. This option is
87 + a shorthand for providing the explicit tag refspec along with
88 + `--prune`, see the discussion about that in its documentation.
89 +
90 See the PRUNING section below for more details.
91
Documentation/git-fetch.txt
+47
@@ -148,6 +148,53 @@ So be careful when using this with a refspec like
148 `refs/tags/*:refs/tags/*`, or any other refspec which might map
149 references from multiple remotes to the same local namespace.
150
151 +Since keeping up-to-date with both branches and tags on the remote is
152 +a common use-case the `--prune-tags` option can be supplied along with
153 +`--prune` to prune local tags that don't exist on the remote, and
154 +force-update those tags that differ. Tag pruning can also be enabled
155 +with `fetch.pruneTags` or `remote.<name>.pruneTags` in the config. See
156 +linkgit:git-config[1].
157 +
158 +The `--prune-tags` option is equivalent to having
159 +`refs/tags/*:refs/tags/*` declared in the refspecs of the remote. This
160 +can lead to some seemingly strange interactions:
161 +
162 +------------------------------------------------
163 +# These both fetch tags
164 +$ git fetch --no-tags origin 'refs/tags/*:refs/tags/*'
165 +$ git fetch --no-tags --prune-tags origin
166 +------------------------------------------------
167 +
168 +The reason it doesn't error out when provided without `--prune` or its
169 +config versions is for flexibility of the configured versions, and to
170 +maintain a 1=1 mapping between what the command line flags do, and
171 +what the configuration versions do.
172 +
173 +It's reasonable to e.g. configure `fetch.pruneTags=true` in
174 +`~/.gitconfig` to have tags pruned whenever `git fetch --prune` is
175 +run, without making every invocation of `git fetch` without `--prune`
176 +an error.
177 +
178 +Another special case of `--prune-tags` is that
179 +`refs/tags/*:refs/tags/*` will not be implicitly provided if an URL is
180 +being fetched. I.e.:
181 +
182 +------------------------------------------------
183 +$ git fetch <url> --prune --prune-tags
184 +------------------------------------------------
185 +
186 +Will prune no tags, as opposed to:
187 +
188 +------------------------------------------------
189 +$ git fetch origin --prune --prune-tags
190 +------------------------------------------------
191 +
192 +To prune tags given a URL supply the refspec explicitly:
193 +
194 +------------------------------------------------
195 +$ git fetch <url> --prune 'refs/tags/*:refs/tags/*'
196 +------------------------------------------------
197 +
198 OUTPUT
199 ------
200
builtin/fetch.c
+29 -3
@@ -38,6 +38,10 @@ static int fetch_prune_config = -1; /* unspecified */
38 static int prune = -1; /* unspecified */
39 #define PRUNE_BY_DEFAULT 0 /* do we prune by default? */
40
41 +static int fetch_prune_tags_config = -1; /* unspecified */
42 +static int prune_tags = -1; /* unspecified */
43 +#define PRUNE_TAGS_BY_DEFAULT 0 /* do we prune tags by default? */
44 +
45 static int all, append, dry_run, force, keep, multiple, update_head_ok, verbosity, deepen_relative;
46 static int progress = -1;
47 static int tags = TAGS_DEFAULT, unshallow, update_shallow, deepen;
@@ -64,6 +68,11 @@ static int git_fetch_config(const char *k, const char *v, void *cb)
68 return 0;
69 }
70
71 + if (!strcmp(k, "fetch.prunetags")) {
72 + fetch_prune_tags_config = git_config_bool(k, v);
73 + return 0;
74 + }
75 +
76 if (!strcmp(k, "submodule.recurse")) {
77 int r = git_config_bool(k, v) ?
78 RECURSE_SUBMODULES_ON : RECURSE_SUBMODULES_OFF;
@@ -126,6 +135,8 @@ static struct option builtin_fetch_options[] = {
135 N_("number of submodules fetched in parallel")),
136 OPT_BOOL('p', "prune", &prune,
137 N_("prune remote-tracking branches no longer on remote")),
138 + OPT_BOOL('P', "prune-tags", &prune_tags,
139 + N_("prune local tags no longer on remote and clobber changed tags")),
140 { OPTION_CALLBACK, 0, "recurse-submodules", &recurse_submodules, N_("on-demand"),
141 N_("control recursive fetching of submodules"),
142 PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules },
@@ -1212,6 +1223,8 @@ static void add_options_to_argv(struct argv_array *argv)
1223 argv_array_push(argv, "--dry-run");
1224 if (prune != -1)
1225 argv_array_push(argv, prune ? "--prune" : "--no-prune");
1226 + if (prune_tags != -1)
1227 + argv_array_push(argv, prune_tags ? "--prune-tags" : "--no-prune-tags");
1228 if (update_head_ok)
1229 argv_array_push(argv, "--update-head-ok");
1230 if (force)
@@ -1265,7 +1278,7 @@ static int fetch_multiple(struct string_list *list)
1278 return result;
1279 }
1280
1268 -static int fetch_one(struct remote *remote, int argc, const char **argv)
1281 +static int fetch_one(struct remote *remote, int argc, const char **argv, int prune_tags_ok)
1282 {
1283 static const char **refs = NULL;
1284 struct refspec *refspec;
@@ -1288,6 +1301,19 @@ static int fetch_one(struct remote *remote, int argc, const char **argv)
1301 prune = PRUNE_BY_DEFAULT;
1302 }
1303
1304 + if (prune_tags < 0) {
1305 + /* no command line request */
1306 + if (0 <= remote->prune_tags)
1307 + prune_tags = remote->prune_tags;
1308 + else if (0 <= fetch_prune_tags_config)
1309 + prune_tags = fetch_prune_tags_config;
1310 + else
1311 + prune_tags = PRUNE_TAGS_BY_DEFAULT;
1312 + }
1313 +
1314 + if (prune_tags_ok && prune_tags && remote_is_configured(remote, 0))
1315 + add_prune_tags_to_fetch_refspec(remote);
1316 +
1317 if (argc > 0) {
1318 int j = 0;
1319 int i;
@@ -1368,7 +1394,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
1394 } else if (argc == 0) {
1395 /* No arguments -- use default remote */
1396 remote = remote_get(NULL);
1371 - result = fetch_one(remote, argc, argv);
1397 + result = fetch_one(remote, argc, argv, 1);
1398 } else if (multiple) {
1399 /* All arguments are assumed to be remotes or groups */
1400 for (i = 0; i < argc; i++)
@@ -1386,7 +1412,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
1412 } else {
1413 /* Zero or one remotes */
1414 remote = remote_get(argv[0]);
1389 - result = fetch_one(remote, argc-1, argv+1);
1415 + result = fetch_one(remote, argc-1, argv+1, argc == 1);
1416 }
1417 }
1418
contrib/completion/git-completion.bash
+1 -1
@@ -1468,7 +1468,7 @@ __git_fetch_recurse_submodules="yes on-demand no"
1468 __git_fetch_options="
1469 --quiet --verbose --append --upload-pack --force --keep --depth=
1470 --tags --no-tags --all --prune --dry-run --recurse-submodules=
1471 - --unshallow --update-shallow
1471 + --unshallow --update-shallow --prune-tags
1472 "
1473
1474 _git_fetch ()
remote.c
+14
@@ -104,6 +104,17 @@ static void add_fetch_refspec(struct remote *remote, const char *ref)
104 remote->fetch_refspec[remote->fetch_refspec_nr++] = ref;
105 }
106
107 +void add_prune_tags_to_fetch_refspec(struct remote *remote)
108 +{
109 + int nr = remote->fetch_refspec_nr;
110 + int bufsize = nr + 1;
111 + int size = sizeof(struct refspec);
112 +
113 + remote->fetch = xrealloc(remote->fetch, size * bufsize);
114 + memcpy(&remote->fetch[nr], tag_refspec, size);
115 + add_fetch_refspec(remote, xstrdup(TAG_REFSPEC));
116 +}
117 +
118 static void add_url(struct remote *remote, const char *url)
119 {
120 ALLOC_GROW(remote->url, remote->url_nr + 1, remote->url_alloc);
@@ -174,6 +185,7 @@ static struct remote *make_remote(const char *name, int len)
185
186 ret = xcalloc(1, sizeof(struct remote));
187 ret->prune = -1; /* unspecified */
188 + ret->prune_tags = -1; /* unspecified */
189 ALLOC_GROW(remotes, remotes_nr + 1, remotes_alloc);
190 remotes[remotes_nr++] = ret;
191 ret->name = xstrndup(name, len);
@@ -392,6 +404,8 @@ static int handle_config(const char *key, const char *value, void *cb)
404 remote->skip_default_update = git_config_bool(key, value);
405 else if (!strcmp(subkey, "prune"))
406 remote->prune = git_config_bool(key, value);
407 + else if (!strcmp(subkey, "prunetags"))
408 + remote->prune_tags = git_config_bool(key, value);
409 else if (!strcmp(subkey, "url")) {
410 const char *v;
411 if (git_config_string(&v, key, value))
remote.h
+3
@@ -47,6 +47,7 @@ struct remote {
47 int skip_default_update;
48 int mirror;
49 int prune;
50 + int prune_tags;
51
52 const char *receivepack;
53 const char *uploadpack;
@@ -299,4 +300,6 @@ void apply_push_cas(struct push_cas_option *, struct remote *, struct ref *);
300
301 #define TAG_REFSPEC "refs/tags/*:refs/tags/*"
302
303 +void add_prune_tags_to_fetch_refspec(struct remote *remote);
304 +
305 #endif
t/t5510-fetch.sh
+70
@@ -589,6 +589,15 @@ test_configured_prune_type () {
589 new_cmdline=$(printf "%s" "$cmdline" | perl -pe 's[origin(?!/)]["'"$remote_url"'"]g')
590 fi
591
592 + if test "$fetch_prune_tags" = 'true' ||
593 + test "$remote_origin_prune_tags" = 'true'
594 + then
595 + if ! printf '%s' "$cmdline\n" | grep -q refs/remotes/origin/
596 + then
597 + new_cmdline="$new_cmdline refs/tags/*:refs/tags/*"
598 + fi
599 + fi
600 +
601 cmdline="$new_cmdline"
602 fi
603
@@ -702,6 +711,67 @@ test_configured_prune true true unset unset kept pruned \
711 test_configured_prune true true unset unset pruned pruned \
712 "--prune origin refs/tags/*:refs/tags/* +refs/heads/*:refs/remotes/origin/*"
713
714 +# --prune-tags on its own does nothing, needs --prune as well, same
715 +# for for fetch.pruneTags without fetch.prune
716 +test_configured_prune unset unset unset unset kept kept "--prune-tags"
717 +test_configured_prune unset unset true unset kept kept ""
718 +test_configured_prune unset unset unset true kept kept ""
719 +
720 +# These will prune the tags
721 +test_configured_prune unset unset unset unset pruned pruned "--prune --prune-tags"
722 +test_configured_prune true unset true unset pruned pruned ""
723 +test_configured_prune unset true unset true pruned pruned ""
724 +
725 +# remote.<name>.pruneTags overrides fetch.pruneTags, just like
726 +# remote.<name>.prune overrides fetch.prune if set.
727 +test_configured_prune true unset true unset pruned pruned ""
728 +test_configured_prune false true false true pruned pruned ""
729 +test_configured_prune true false true false kept kept ""
730 +
731 +# When --prune-tags is supplied it's ignored if an explicit refspec is
732 +# given, same for the configuration options.
733 +test_configured_prune unset unset unset unset pruned kept \
734 + "--prune --prune-tags origin +refs/heads/*:refs/remotes/origin/*"
735 +test_configured_prune unset unset true unset pruned kept \
736 + "--prune origin +refs/heads/*:refs/remotes/origin/*"
737 +test_configured_prune unset unset unset true pruned kept \
738 + "--prune origin +refs/heads/*:refs/remotes/origin/*"
739 +
740 +# Pruning that also takes place if a file:// url replaces a named
741 +# remote, with the exception of --prune-tags on the command-line
742 +# (arbitrary limitation).
743 +#
744 +# However, because there's no implicit
745 +# +refs/heads/*:refs/remotes/origin/* refspec and supplying it on the
746 +# command-line negates --prune-tags, the branches will not be pruned.
747 +test_configured_prune_type unset unset unset unset kept kept "origin --prune-tags" "name"
748 +test_configured_prune_type unset unset unset unset kept kept "origin --prune-tags" "link"
749 +test_configured_prune_type unset unset unset unset pruned pruned "origin --prune --prune-tags" "name"
750 +test_configured_prune_type unset unset unset unset kept kept "origin --prune --prune-tags" "link"
751 +test_configured_prune_type unset unset unset unset pruned pruned "--prune --prune-tags origin" "name"
752 +test_configured_prune_type unset unset unset unset kept kept "--prune --prune-tags origin" "link"
753 +test_configured_prune_type unset unset true unset pruned pruned "--prune origin" "name"
754 +test_configured_prune_type unset unset true unset kept pruned "--prune origin" "link"
755 +test_configured_prune_type unset unset unset true pruned pruned "--prune origin" "name"
756 +test_configured_prune_type unset unset unset true kept pruned "--prune origin" "link"
757 +test_configured_prune_type true unset true unset pruned pruned "origin" "name"
758 +test_configured_prune_type true unset true unset kept pruned "origin" "link"
759 +test_configured_prune_type unset true true unset pruned pruned "origin" "name"
760 +test_configured_prune_type unset true true unset kept pruned "origin" "link"
761 +test_configured_prune_type unset true unset true pruned pruned "origin" "name"
762 +test_configured_prune_type unset true unset true kept pruned "origin" "link"
763 +
764 +# Interaction between --prune-tags and no "fetch" config in the remote
765 +# at all.
766 +test_expect_success 'remove remote.origin.fetch "one"' '
767 + (
768 + cd one &&
769 + git config --unset-all remote.origin.fetch
770 + )
771 +'
772 +test_configured_prune_type unset unset unset unset kept pruned "origin --prune --prune-tags" "name"
773 +test_configured_prune_type unset unset unset unset kept kept "origin --prune --prune-tags" "link"
774 +
775 test_expect_success 'all boundary commits are excluded' '
776 test_commit base &&
777 test_commit oneside &&