@matheusbsbs / matheus-bagliano / commits / dd781e3856

fetch: introduce machine-parseable "porcelain" output format

The output of git-fetch(1) is obviously designed for consumption by users, only: we neatly columnize data, we abbreviate reference names, we print neat arrows and we don't provide information about actual object IDs that have changed. This makes the output format basically unusable in the context of scripted invocations of git-fetch(1) that want to learn about the exact changes that the command performs. Introduce a new machine-parseable "porcelain" output format that is supposed to fix this shortcoming. This output format is intended to provide information about every reference that is about to be updated, the old object ID that the reference has been pointing to and the new object ID it will be updated to. Furthermore, the output format provides the same flags as the human-readable format to indicate basic conditions for each reference update like whether it was a fast-forward update, a branch deletion, a rejected update or others. The output format is quite simple: ``` <flag> <old-object-id> <new-object-id> <local-reference>\n ``` We assume two conditions which are generally true: - The old and new object IDs have fixed known widths and cannot contain spaces. - References cannot contain newlines. With these assumptions, the output format becomes unambiguously parseable. Furthermore, given that this output is designed to be consumed by scripts, the machine-readable data is printed to stdout instead of stderr like the human-readable output is. This is mostly done so that other data printed to stderr, like error messages or progress meters, don't interfere with the parseable data. A notable ommission here is that the output format does not include the remote from which a reference was fetched, which might be important information especially in the context of multi-remote fetches. But as such a format would require us to print the remote for every single reference update due to parallelizable fetches it feels wasteful for the most likely usecase, which is when fetching from a single remote. In a similar spirit, a second restriction is that this cannot be used with `--recurse-submodules`. This is because any reference updates would be ambiguous without also printing the repository in which the update happens. Considering that both multi-remote and submodule fetches are user-facing features, using them in conjunction with `--porcelain` that is intended for scripting purposes is likely not going to be useful in the majority of cases. With that in mind these restrictions feel acceptable. If usecases for either of these come up in the future though it is easy enough to add a new "porcelain-v2" format that adds this information. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Patrick Steinhardt committed May 10, 2023 at 14:34 UTC dd781e3856bccd3793122f7f4e604e5a89ae517d

4 files changed +244 -19

Documentation/fetch-options.txt

@@ -78,6 +78,13 @@ linkgit:git-config[1].

78 --dry-run::

79 Show what would be done, without making any changes.

81 +--porcelain::

82 + Print the output to standard output in an easy-to-parse format for

83 + scripts. See section OUTPUT in linkgit:git-fetch[1] for details.

84 ++

85 +This is incompatible with `--recurse-submodules=[yes|on-demand]` and takes

86 +precedence over the `fetch.output` config option.

87 +

88 ifndef::git-pull[]

89 --[no-]write-fetch-head::

90 Write the list of remote refs fetched in the `FETCH_HEAD`

Documentation/git-fetch.txt

@@ -204,6 +204,15 @@ representing the status of a single ref. Each line is of the form:

204 <flag> <summary> <from> -> <to> [<reason>]

205 -------------------------------

206

207 +When using `--porcelain`, the output format is intended to be

208 +machine-parseable. In contrast to the human-readable output formats it

209 +thus prints to standard output instead of standard error. Each line is

210 +of the form:

211 +

212 +-------------------------------

213 +<flag> <old-object-id> <new-object-id> <local-reference>

214 +-------------------------------

215 +

216 The status of up-to-date refs is shown only if the --verbose option is

217 used.

218

builtin/fetch.c

+69 -19

@@ -54,6 +54,7 @@ enum display_format {

54 DISPLAY_FORMAT_UNKNOWN = 0,

55 DISPLAY_FORMAT_FULL,

56 DISPLAY_FORMAT_COMPACT,

57 + DISPLAY_FORMAT_PORCELAIN,

58 };

60 struct display_state {

@@ -756,6 +757,9 @@ static void display_state_init(struct display_state *display_state, struct ref *

757 display_state->refcol_width = refcol_width(ref_map,

758 display_state->format == DISPLAY_FORMAT_COMPACT);

759 break;

760 + case DISPLAY_FORMAT_PORCELAIN:

761 + /* We don't need to precompute anything here. */

762 + break;

763 default:

764 BUG("unexpected display format %d", display_state->format);

765 }

@@ -826,8 +830,12 @@ static void print_compact(struct display_state *display_state,

830 static void display_ref_update(struct display_state *display_state, char code,

831 const char *summary, const char *error,

832 const char *remote, const char *local,

833 + const struct object_id *old_oid,

834 + const struct object_id *new_oid,

835 int summary_width)

836 {

837 + FILE *f = stderr;

838 +

839 if (verbosity < 0)

840 return;

841

@@ -860,12 +868,17 @@ static void display_ref_update(struct display_state *display_state, char code,

868

869 break;

870 }

871 + case DISPLAY_FORMAT_PORCELAIN:

872 + strbuf_addf(&display_state->buf, "%c %s %s %s", code,

873 + oid_to_hex(old_oid), oid_to_hex(new_oid), local);

874 + f = stdout;

875 + break;

876 default:

877 BUG("unexpected display format %d", display_state->format);

878 };

879 strbuf_addch(&display_state->buf, '\n');

880

868 - fputs(display_state->buf.buf, stderr);

881 + fputs(display_state->buf.buf, f);

882 }

883

884 static int update_local_ref(struct ref *ref,

@@ -883,7 +896,8 @@ static int update_local_ref(struct ref *ref,

896 if (oideq(&ref->old_oid, &ref->new_oid)) {

897 if (verbosity > 0)

898 display_ref_update(display_state, '=', _("[up to date]"), NULL,

886 - remote_ref->name, ref->name, summary_width);

899 + remote_ref->name, ref->name,

900 + &ref->old_oid, &ref->new_oid, summary_width);

901 return 0;

902 }

903

@@ -896,7 +910,8 @@ static int update_local_ref(struct ref *ref,

910 */

911 display_ref_update(display_state, '!', _("[rejected]"),

912 _("can't fetch into checked-out branch"),

899 - remote_ref->name, ref->name, summary_width);

913 + remote_ref->name, ref->name,

914 + &ref->old_oid, &ref->new_oid, summary_width);

915 return 1;

916 }

917

@@ -907,12 +922,14 @@ static int update_local_ref(struct ref *ref,

922 r = s_update_ref("updating tag", ref, transaction, 0);

923 display_ref_update(display_state, r ? '!' : 't', _("[tag update]"),

924 r ? _("unable to update local ref") : NULL,

910 - remote_ref->name, ref->name, summary_width);

925 + remote_ref->name, ref->name,

926 + &ref->old_oid, &ref->new_oid, summary_width);

927 return r;

928 } else {

929 display_ref_update(display_state, '!', _("[rejected]"),

930 _("would clobber existing tag"),

915 - remote_ref->name, ref->name, summary_width);

931 + remote_ref->name, ref->name,

932 + &ref->old_oid, &ref->new_oid, summary_width);

933 return 1;

934 }

935 }

@@ -945,7 +962,8 @@ static int update_local_ref(struct ref *ref,

962 r = s_update_ref(msg, ref, transaction, 0);

963 display_ref_update(display_state, r ? '!' : '*', what,

964 r ? _("unable to update local ref") : NULL,

948 - remote_ref->name, ref->name, summary_width);

965 + remote_ref->name, ref->name,

966 + &ref->old_oid, &ref->new_oid, summary_width);

967 return r;

968 }

969

@@ -968,7 +986,8 @@ static int update_local_ref(struct ref *ref,

986 r = s_update_ref("fast-forward", ref, transaction, 1);

987 display_ref_update(display_state, r ? '!' : ' ', quickref.buf,

988 r ? _("unable to update local ref") : NULL,

971 - remote_ref->name, ref->name, summary_width);

989 + remote_ref->name, ref->name,

990 + &ref->old_oid, &ref->new_oid, summary_width);

991 strbuf_release(&quickref);

992 return r;

993 } else if (force || ref->force) {

@@ -980,12 +999,14 @@ static int update_local_ref(struct ref *ref,

999 r = s_update_ref("forced-update", ref, transaction, 1);

1000 display_ref_update(display_state, r ? '!' : '+', quickref.buf,

1001 r ? _("unable to update local ref") : _("forced update"),

983 - remote_ref->name, ref->name, summary_width);

1002 + remote_ref->name, ref->name,

1003 + &ref->old_oid, &ref->new_oid, summary_width);

1004 strbuf_release(&quickref);

1005 return r;

1006 } else {

1007 display_ref_update(display_state, '!', _("[rejected]"), _("non-fast-forward"),

988 - remote_ref->name, ref->name, summary_width);

1008 + remote_ref->name, ref->name,

1009 + &ref->old_oid, &ref->new_oid, summary_width);

1010 return 1;

1011 }

1012 }

@@ -1226,7 +1247,9 @@ static int store_updated_refs(struct display_state *display_state,

1247 display_ref_update(display_state, '*',

1248 *kind ? kind : "branch", NULL,

1249 rm->name,

1229 - "FETCH_HEAD", summary_width);

1250 + "FETCH_HEAD",

1251 + &rm->new_oid, &rm->old_oid,

1252 + summary_width);

1253 }

1254 }

1255 }

@@ -1366,6 +1389,7 @@ static int prune_refs(struct display_state *display_state,

1389 for (ref = stale_refs; ref; ref = ref->next) {

1390 display_ref_update(display_state, '-', _("[deleted]"), NULL,

1391 _("(none)"), ref->name,

1392 + &ref->new_oid, &ref->old_oid,

1393 summary_width);

1394 warn_dangling_symref(stderr, dangling_msg, ref->name);

1395 }

@@ -1798,7 +1822,8 @@ static int add_remote_or_group(const char *name, struct string_list *list)

1822 return 1;

1823 }

1824

1801 -static void add_options_to_argv(struct strvec *argv)

1825 +static void add_options_to_argv(struct strvec *argv,

1826 + enum display_format format)

1827 {

1828 if (dry_run)

1829 strvec_push(argv, "--dry-run");

@@ -1834,6 +1859,8 @@ static void add_options_to_argv(struct strvec *argv)

1859 strvec_push(argv, "--ipv6");

1860 if (!write_fetch_head)

1861 strvec_push(argv, "--no-write-fetch-head");

1862 + if (format == DISPLAY_FORMAT_PORCELAIN)

1863 + strvec_pushf(argv, "--porcelain");

1864 }

1865

1866 /* Fetch multiple remotes in parallel */

@@ -1842,6 +1869,7 @@ struct parallel_fetch_state {

1869 const char **argv;

1870 struct string_list *remotes;

1871 int next, result;

1872 + enum display_format format;

1873 };

1874

1875 static int fetch_next_remote(struct child_process *cp,

@@ -1861,7 +1889,7 @@ static int fetch_next_remote(struct child_process *cp,

1889 strvec_push(&cp->args, remote);

1890 cp->git_cmd = 1;

1891

1864 - if (verbosity >= 0)

1892 + if (verbosity >= 0 && state->format != DISPLAY_FORMAT_PORCELAIN)

1893 printf(_("Fetching %s\n"), remote);

1894

1895 return 1;

@@ -1893,7 +1921,8 @@ static int fetch_finished(int result, struct strbuf *out,

1921 return 0;

1922 }

1923

1896 -static int fetch_multiple(struct string_list *list, int max_children)

1924 +static int fetch_multiple(struct string_list *list, int max_children,

1925 + enum display_format format)

1926 {

1927 int i, result = 0;

1928 struct strvec argv = STRVEC_INIT;

@@ -1911,10 +1940,10 @@ static int fetch_multiple(struct string_list *list, int max_children)

1940 strvec_pushl(&argv, "-c", "fetch.bundleURI=",

1941 "fetch", "--append", "--no-auto-gc",

1942 "--no-write-commit-graph", NULL);

1914 - add_options_to_argv(&argv);

1943 + add_options_to_argv(&argv, format);

1944

1945 if (max_children != 1 && list->nr != 1) {

1917 - struct parallel_fetch_state state = { argv.v, list, 0, 0 };

1946 + struct parallel_fetch_state state = { argv.v, list, 0, 0, format };

1947 const struct run_process_parallel_opts opts = {

1948 .tr2_category = "fetch",

1949 .tr2_label = "parallel/fetch",

@@ -1938,7 +1967,7 @@ static int fetch_multiple(struct string_list *list, int max_children)

1967

1968 strvec_pushv(&cmd.args, argv.v);

1969 strvec_push(&cmd.args, name);

1941 - if (verbosity >= 0)

1970 + if (verbosity >= 0 && format != DISPLAY_FORMAT_PORCELAIN)

1971 printf(_("Fetching %s\n"), name);

1972 cmd.git_cmd = 1;

1973 if (run_command(&cmd)) {

@@ -2089,6 +2118,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)

2118 int fetch_write_commit_graph = -1;

2119 int stdin_refspecs = 0;

2120 int negotiate_only = 0;

2121 + int porcelain = 0;

2122 int i;

2123

2124 struct option builtin_fetch_options[] = {

@@ -2123,6 +2153,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)

2153 PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),

2154 OPT_BOOL(0, "dry-run", &dry_run,

2155 N_("dry run")),

2156 + OPT_BOOL(0, "porcelain", &porcelain, N_("machine-readable output")),

2157 OPT_BOOL(0, "write-fetch-head", &write_fetch_head,

2158 N_("write fetched references to the FETCH_HEAD file")),

2159 OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),

@@ -2228,6 +2259,26 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)

2259 fetch_config_from_gitmodules(sfjc, rs);

2260 }

2261

2262 +

2263 + if (porcelain) {

2264 + switch (recurse_submodules_cli) {

2265 + case RECURSE_SUBMODULES_OFF:

2266 + case RECURSE_SUBMODULES_DEFAULT:

2267 + /*

2268 + * Reference updates in submodules would be ambiguous

2269 + * in porcelain mode, so we reject this combination.

2270 + */

2271 + recurse_submodules = RECURSE_SUBMODULES_OFF;

2272 + break;

2273 +

2274 + default:

2275 + die(_("options '%s' and '%s' cannot be used together"),

2276 + "--porcelain", "--recurse-submodules");

2277 + }

2278 +

2279 + config.display_format = DISPLAY_FORMAT_PORCELAIN;

2280 + }

2281 +

2282 if (negotiate_only && !negotiation_tip.nr)

2283 die(_("--negotiate-only needs one or more --negotiation-tip=*"));

2284

@@ -2347,10 +2398,9 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)

2398 max_children = fetch_parallel_config;

2399

2400 /* TODO should this also die if we have a previous partial-clone? */

2350 - result = fetch_multiple(&list, max_children);

2401 + result = fetch_multiple(&list, max_children, config.display_format);

2402 }

2403

2353 -

2404 /*

2405 * This is only needed after fetch_one(), which does not fetch

2406 * submodules by itself.

@@ -2369,7 +2419,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)

2419 if (max_children < 0)

2420 max_children = fetch_parallel_config;

2421

2372 - add_options_to_argv(&options);

2422 + add_options_to_argv(&options, config.display_format);

2423 result = fetch_submodules(the_repository,

2424 &options,

2425 submodule_prefix,

t/t5574-fetch-output.sh

+159

@@ -61,6 +61,141 @@ test_expect_success 'fetch compact output' '

61 test_cmp expect actual

62 '

64 +test_expect_success 'fetch porcelain output' '

65 + test_when_finished "rm -rf porcelain" &&

66 +

67 + # Set up a bunch of references that we can use to demonstrate different

68 + # kinds of flag symbols in the output format.

69 + MAIN_OLD=$(git rev-parse HEAD) &&

70 + git branch "fast-forward" &&

71 + git branch "deleted-branch" &&

72 + git checkout -b force-updated &&

73 + test_commit --no-tag force-update-old &&

74 + FORCE_UPDATED_OLD=$(git rev-parse HEAD) &&

75 + git checkout main &&

76 +

77 + # Clone and pre-seed the repositories. We fetch references into two

78 + # namespaces so that we can test that rejected and force-updated

79 + # references are reported properly.

80 + refspecs="refs/heads/*:refs/unforced/* +refs/heads/*:refs/forced/*" &&

81 + git clone . porcelain &&

82 + git -C porcelain fetch origin $refspecs &&

83 +

84 + # Now that we have set up the client repositories we can change our

85 + # local references.

86 + git branch new-branch &&

87 + git branch -d deleted-branch &&

88 + git checkout fast-forward &&

89 + test_commit --no-tag fast-forward-new &&

90 + FAST_FORWARD_NEW=$(git rev-parse HEAD) &&

91 + git checkout force-updated &&

92 + git reset --hard HEAD~ &&

93 + test_commit --no-tag force-update-new &&

94 + FORCE_UPDATED_NEW=$(git rev-parse HEAD) &&

95 +

96 + cat >expect <<-EOF &&

97 + - $MAIN_OLD $ZERO_OID refs/forced/deleted-branch

98 + - $MAIN_OLD $ZERO_OID refs/unforced/deleted-branch

99 + $MAIN_OLD $FAST_FORWARD_NEW refs/unforced/fast-forward

100 + ! $FORCE_UPDATED_OLD $FORCE_UPDATED_NEW refs/unforced/force-updated

101 + * $ZERO_OID $MAIN_OLD refs/unforced/new-branch

102 + $MAIN_OLD $FAST_FORWARD_NEW refs/forced/fast-forward

103 + + $FORCE_UPDATED_OLD $FORCE_UPDATED_NEW refs/forced/force-updated

104 + * $ZERO_OID $MAIN_OLD refs/forced/new-branch

105 + $MAIN_OLD $FAST_FORWARD_NEW refs/remotes/origin/fast-forward

106 + + $FORCE_UPDATED_OLD $FORCE_UPDATED_NEW refs/remotes/origin/force-updated

107 + * $ZERO_OID $MAIN_OLD refs/remotes/origin/new-branch

108 + EOF

109 +

110 + # Execute a dry-run fetch first. We do this to assert that the dry-run

111 + # and non-dry-run fetches produces the same output. Execution of the

112 + # fetch is expected to fail as we have a rejected reference update.

113 + test_must_fail git -C porcelain fetch \

114 + --porcelain --dry-run --prune origin $refspecs >actual &&

115 + test_cmp expect actual &&

116 +

117 + # And now we perform a non-dry-run fetch.

118 + test_must_fail git -C porcelain fetch \

119 + --porcelain --prune origin $refspecs >actual 2>stderr &&

120 + test_cmp expect actual &&

121 + test_must_be_empty stderr

122 +'

123 +

124 +test_expect_success 'fetch porcelain with multiple remotes' '

125 + test_when_finished "rm -rf porcelain" &&

126 +

127 + git switch --create multiple-remotes &&

128 + git clone . porcelain &&

129 + git -C porcelain remote add second-remote "$PWD" &&

130 + git -C porcelain fetch second-remote &&

131 +

132 + test_commit --no-tag multi-commit &&

133 + old_commit=$(git rev-parse HEAD~) &&

134 + new_commit=$(git rev-parse HEAD) &&

135 +

136 + cat >expect <<-EOF &&

137 + $old_commit $new_commit refs/remotes/origin/multiple-remotes

138 + $old_commit $new_commit refs/remotes/second-remote/multiple-remotes

139 + EOF

140 +

141 + git -C porcelain fetch --porcelain --all >actual 2>stderr &&

142 + test_cmp expect actual &&

143 + test_must_be_empty stderr

144 +'

145 +

146 +test_expect_success 'fetch porcelain refuses to work with submodules' '

147 + test_when_finished "rm -rf porcelain" &&

148 +

149 + cat >expect <<-EOF &&

150 + fatal: options ${SQ}--porcelain${SQ} and ${SQ}--recurse-submodules${SQ} cannot be used together

151 + EOF

152 +

153 + git init porcelain &&

154 + test_must_fail git -C porcelain fetch --porcelain --recurse-submodules=yes 2>stderr &&

155 + test_cmp expect stderr &&

156 +

157 + test_must_fail git -C porcelain fetch --porcelain --recurse-submodules=on-demand 2>stderr &&

158 + test_cmp expect stderr

159 +'

160 +

161 +test_expect_success 'fetch porcelain overrides fetch.output config' '

162 + test_when_finished "rm -rf porcelain" &&

163 +

164 + git switch --create config-override &&

165 + git clone . porcelain &&

166 + test_commit new-commit &&

167 + old_commit=$(git rev-parse HEAD~) &&

168 + new_commit=$(git rev-parse HEAD) &&

169 +

170 + cat >expect <<-EOF &&

171 + $old_commit $new_commit refs/remotes/origin/config-override

172 + * $ZERO_OID $new_commit refs/tags/new-commit

173 + EOF

174 +

175 + git -C porcelain -c fetch.output=compact fetch --porcelain >stdout 2>stderr &&

176 + test_must_be_empty stderr &&

177 + test_cmp expect stdout

178 +'

179 +

180 +test_expect_success 'fetch --no-porcelain overrides previous --porcelain' '

181 + test_when_finished "rm -rf no-porcelain" &&

182 +

183 + git switch --create no-porcelain &&

184 + git clone . no-porcelain &&

185 + test_commit --no-tag no-porcelain &&

186 + old_commit=$(git rev-parse --short HEAD~) &&

187 + new_commit=$(git rev-parse --short HEAD) &&

188 +

189 + cat >expect <<-EOF &&

190 + From $(test-tool path-utils real_path .)/.

191 + $old_commit..$new_commit no-porcelain -> origin/no-porcelain

192 + EOF

193 +

194 + git -C no-porcelain fetch --porcelain --no-porcelain >stdout 2>stderr &&

195 + test_cmp expect stderr &&

196 + test_must_be_empty stdout

197 +'

198 +

199 test_expect_success 'fetch output with HEAD' '

200 test_when_finished "rm -rf head" &&

201 git clone . head &&

@@ -90,6 +225,30 @@ test_expect_success 'fetch output with HEAD' '

225 test_cmp expect actual.err

226 '

227

228 +test_expect_success 'fetch porcelain output with HEAD' '

229 + test_when_finished "rm -rf head" &&

230 + git clone . head &&

231 + COMMIT_ID=$(git rev-parse HEAD) &&

232 +

233 + git -C head fetch --porcelain --dry-run origin HEAD >actual &&

234 + cat >expect <<-EOF &&

235 + * $ZERO_OID $COMMIT_ID FETCH_HEAD

236 + EOF

237 + test_cmp expect actual &&

238 +

239 + git -C head fetch --porcelain origin HEAD >actual &&

240 + test_cmp expect actual &&

241 +

242 + git -C head fetch --porcelain --dry-run origin HEAD:foo >actual &&

243 + cat >expect <<-EOF &&

244 + * $ZERO_OID $COMMIT_ID refs/heads/foo

245 + EOF

246 + test_cmp expect actual &&

247 +

248 + git -C head fetch --porcelain origin HEAD:foo >actual &&

249 + test_cmp expect actual

250 +'

251 +

252 test_expect_success 'fetch output with object ID' '

253 test_when_finished "rm -rf object-id" &&

254 git clone . object-id &&