@matheusbsbs / matheus-bagliano / commits / 63e0c8b364
Fork 0

Support RFC 2822 date parsing in fast-import.

Since some frontends may be working with source material where the dates are only readily available as RFC 2822 strings, it is more friendly if fast-import exposes Git's parse_date() function to handle the conversion. This way the frontend doesn't need to perform the parsing itself. The new --date-format option to fast-import can be used by a frontend to select which format it will supply date strings in. The default is the standard `raw` Git format, which fast-import has always supported. Format rfc2822 can be used to activate the parse_date() function instead. Because fast-import could also be useful for creating new, current commits, the format `now` is also supported to generate the current system timestamp. The implementation of `now` is a trivial call to datestamp(), but is actually a whole whopping 3 lines so that fast-import can verify the frontend really meant `now`. As part of this change I have added validation of the `raw` date format. Prior to this change fast-import would accept anything in a `committer` command, even if it was seriously malformed. Now fast-import requires the '> ' near the end of the string and verifies the timestamp is formatted properly. Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Shawn O. Pearce committed Feb 6, 2007 at 14:58 UTC 63e0c8b364e334fc7cc975edf1f16fb4c89594b3
3 files changed +214 -24
Documentation/git-fast-import.txt
+85 -10
@@ -32,6 +32,12 @@ the frontend program in use.
32
33 OPTIONS
34 -------
35 +--date-format=<fmt>::
36 + Specify the type of dates the frontend will supply to
37 + gfi within `author`, `committer` and `tagger` commands.
38 + See ``Date Formats'' below for details about which formats
39 + are supported, and their syntax.
40 +
41 --max-pack-size=<n>::
42 Maximum size of each output packfile, expressed in MiB.
43 The default is 4096 (4 GiB) as that is the maximum allowed
@@ -53,7 +59,6 @@ OPTIONS
59 Frontends can use this file to validate imports after they
60 have been completed.
61
56 -
62 Performance
63 -----------
64 The design of gfi allows it to import large projects in a minimum
@@ -127,6 +132,78 @@ results, such as branch names or file names with leading or trailing
132 spaces in their name, or early termination of gfi when it encounters
133 unexpected input.
134
135 +Date Formats
136 +~~~~~~~~~~~~
137 +The following date formats are supported. A frontend should select
138 +the format it will use for this import by passing the format name
139 +in the `--date-format=<fmt>` command line option.
140 +
141 +`raw`::
142 + This is the Git native format and is `<time> SP <tz>`.
143 + It is also gfi's default format, if `--date-format` was
144 + not specified.
145 ++
146 +The time of the event is specified by `<time>` as the number of
147 +seconds since the UNIX epoch (midnight, Jan 1, 1970, UTC) and is
148 +written as an ASCII decimal integer.
149 ++
150 +The timezone is specified by `<tz>` as a positive or negative offset
151 +from UTC. For example EST (which is typically 5 hours behind GMT)
152 +would be expressed in `<tz>` by ``-0500'' while GMT is ``+0000''.
153 ++
154 +If the timezone is not available in the source material, use
155 +``+0000'', or the most common local timezone. For example many
156 +organizations have a CVS repository which has only ever been accessed
157 +by users who are located in the same location and timezone. In this
158 +case the user's timezone can be easily assumed.
159 ++
160 +Unlike the `rfc2822` format, this format is very strict. Any
161 +variation in formatting will cause gfi to reject the value.
162 +
163 +`rfc2822`::
164 + This is the standard email format as described by RFC 2822.
165 ++
166 +An example value is ``Tue Feb 6 11:22:18 2007 -0500''. The Git
167 +parser is accurate, but a little on the lenient side. Its the
168 +same parser used by gitlink:git-am[1] when applying patches
169 +received from email.
170 ++
171 +Some malformed strings may be accepted as valid dates. In some of
172 +these cases Git will still be able to obtain the correct date from
173 +the malformed string. There are also some types of malformed
174 +strings which Git will parse wrong, and yet consider valid.
175 +Seriously malformed strings will be rejected.
176 ++
177 +If the source material is formatted in RFC 2822 style dates,
178 +the frontend should let gfi handle the parsing and conversion
179 +(rather than attempting to do it itself) as the Git parser has
180 +been well tested in the wild.
181 ++
182 +Frontends should prefer the `raw` format if the source material
183 +is already in UNIX-epoch format, or is easily convertible to
184 +that format, as there is no ambiguity in parsing.
185 +
186 +`now`::
187 + Always use the current time and timezone. The literal
188 + `now` must always be supplied for `<when>`.
189 ++
190 +This is a toy format. The current time and timezone of this system
191 +is always copied into the identity string at the time it is being
192 +created by gfi. There is no way to specify a different time or
193 +timezone.
194 ++
195 +This particular format is supplied as its short to implement and
196 +may be useful to a process that wants to create a new commit
197 +right now, without needing to use a working directory or
198 +gitlink:git-update-index[1].
199 ++
200 +If separate `author` and `committer` commands are used in a `commit`
201 +the timestamps may not match, as the system clock will be polled
202 +twice (once for each command). The only way to ensure that both
203 +author and committer identity information has the same timestamp
204 +is to omit `author` (thus copying from `committer`) or to use a
205 +date format other than `now`.
206 +
207 Commands
208 ~~~~~~~~
209 gfi accepts several commands to update the current repository
@@ -168,8 +245,8 @@ change to the project.
245 ....
246 'commit' SP <ref> LF
247 mark?
171 - ('author' SP <name> SP LT <email> GT SP <time> SP <tz> LF)?
172 - 'committer' SP <name> SP LT <email> GT SP <time> SP <tz> LF
248 + ('author' SP <name> SP LT <email> GT SP <when> LF)?
249 + 'committer' SP <name> SP LT <email> GT SP <when> LF
250 data
251 ('from' SP <committish> LF)?
252 ('merge' SP <committish> LF)?
@@ -222,12 +299,10 @@ the email address from the other fields in the line. Note that
299 `<name>` is free-form and may contain any sequence of bytes, except
300 `LT` and `LF`. It is typically UTF-8 encoded.
301
225 -The time of the change is specified by `<time>` as the number of
226 -seconds since the UNIX epoc (midnight, Jan 1, 1970, UTC) and is
227 -written as an ASCII decimal integer. The committer's
228 -timezone is specified by `<tz>` as a positive or negative offset
229 -from UTC. For example EST (which is typically 5 hours behind GMT)
230 -would be expressed in `<tz>` by ``-0500'' while GMT is ``+0000''.
302 +The time of the change is specified by `<when>` using the date format
303 +that was selected by the `--date-format=<fmt>` command line option.
304 +See ``Date Formats'' above for the set of supported formats, and
305 +their syntax.
306
307 `from`
308 ^^^^^^
@@ -394,7 +469,7 @@ lightweight (non-annotated) tags see the `reset` command below.
469 ....
470 'tag' SP <name> LF
471 'from' SP <committish> LF
397 - 'tagger' SP <name> SP LT <email> GT SP <time> SP <tz> LF
472 + 'tagger' SP <name> SP LT <email> GT SP <when> LF
473 data
474 LF
475 ....
fast-import.c
+93 -14
@@ -17,8 +17,8 @@ Format of STDIN stream:
17
18 new_commit ::= 'commit' sp ref_str lf
19 mark?
20 - ('author' sp name '<' email '>' ts tz lf)?
21 - 'committer' sp name '<' email '>' ts tz lf
20 + ('author' sp name '<' email '>' when lf)?
21 + 'committer' sp name '<' email '>' when lf
22 commit_msg
23 ('from' sp (ref_str | hexsha1 | sha1exp_str | idnum) lf)?
24 ('merge' sp (ref_str | hexsha1 | sha1exp_str | idnum) lf)*
@@ -34,7 +34,7 @@ Format of STDIN stream:
34
35 new_tag ::= 'tag' sp tag_str lf
36 'from' sp (ref_str | hexsha1 | sha1exp_str | idnum) lf
37 - 'tagger' sp name '<' email '>' ts tz lf
37 + 'tagger' sp name '<' email '>' when lf
38 tag_msg;
39 tag_msg ::= data;
40
@@ -88,6 +88,10 @@ Format of STDIN stream:
88 bigint ::= # unsigned integer value, ascii base10 notation;
89 binary_data ::= # file content, not interpreted;
90
91 + when ::= raw_when | rfc2822_when;
92 + raw_when ::= ts sp tz;
93 + rfc2822_when ::= # Valid RFC 2822 date and time;
94 +
95 sp ::= # ASCII space character;
96 lf ::= # ASCII newline (LF) character;
97
@@ -234,6 +238,12 @@ struct hash_list
238 unsigned char sha1[20];
239 };
240
241 +typedef enum {
242 + WHENSPEC_RAW = 1,
243 + WHENSPEC_RFC2822,
244 + WHENSPEC_NOW,
245 +} whenspec_type;
246 +
247 /* Configured limits on output */
248 static unsigned long max_depth = 10;
249 static unsigned long max_packsize = (1LL << 32) - 1;
@@ -294,6 +304,7 @@ static struct tag *first_tag;
304 static struct tag *last_tag;
305
306 /* Input stream parsing */
307 +static whenspec_type whenspec = WHENSPEC_RAW;
308 static struct strbuf command_buf;
309 static uintmax_t next_mark;
310 static struct dbuf new_data;
@@ -1396,6 +1407,64 @@ static void *cmd_data (size_t *size)
1407 return buffer;
1408 }
1409
1410 +static int validate_raw_date(const char *src, char *result, int maxlen)
1411 +{
1412 + const char *orig_src = src;
1413 + char *endp, sign;
1414 +
1415 + strtoul(src, &endp, 10);
1416 + if (endp == src || *endp != ' ')
1417 + return -1;
1418 +
1419 + src = endp + 1;
1420 + if (*src != '-' && *src != '+')
1421 + return -1;
1422 + sign = *src;
1423 +
1424 + strtoul(src + 1, &endp, 10);
1425 + if (endp == src || *endp || (endp - orig_src) >= maxlen)
1426 + return -1;
1427 +
1428 + strcpy(result, orig_src);
1429 + return 0;
1430 +}
1431 +
1432 +static char *parse_ident(const char *buf)
1433 +{
1434 + const char *gt;
1435 + size_t name_len;
1436 + char *ident;
1437 +
1438 + gt = strrchr(buf, '>');
1439 + if (!gt)
1440 + die("Missing > in ident string: %s", buf);
1441 + gt++;
1442 + if (*gt != ' ')
1443 + die("Missing space after > in ident string: %s", buf);
1444 + gt++;
1445 + name_len = gt - buf;
1446 + ident = xmalloc(name_len + 24);
1447 + strncpy(ident, buf, name_len);
1448 +
1449 + switch (whenspec) {
1450 + case WHENSPEC_RAW:
1451 + if (validate_raw_date(gt, ident + name_len, 24) < 0)
1452 + die("Invalid raw date \"%s\" in ident: %s", gt, buf);
1453 + break;
1454 + case WHENSPEC_RFC2822:
1455 + if (parse_date(gt, ident + name_len, 24) < 0)
1456 + die("Invalid rfc2822 date \"%s\" in ident: %s", gt, buf);
1457 + break;
1458 + case WHENSPEC_NOW:
1459 + if (strcmp("now", gt))
1460 + die("Date in ident must be 'now': %s", buf);
1461 + datestamp(ident + name_len, 24);
1462 + break;
1463 + }
1464 +
1465 + return ident;
1466 +}
1467 +
1468 static void cmd_new_blob(void)
1469 {
1470 size_t l;
@@ -1655,11 +1724,11 @@ static void cmd_new_commit(void)
1724 read_next_command();
1725 cmd_mark();
1726 if (!strncmp("author ", command_buf.buf, 7)) {
1658 - author = strdup(command_buf.buf);
1727 + author = parse_ident(command_buf.buf + 7);
1728 read_next_command();
1729 }
1730 if (!strncmp("committer ", command_buf.buf, 10)) {
1662 - committer = strdup(command_buf.buf);
1731 + committer = parse_ident(command_buf.buf + 10);
1732 read_next_command();
1733 }
1734 if (!committer)
@@ -1692,7 +1761,7 @@ static void cmd_new_commit(void)
1761 store_tree(&b->branch_tree);
1762 hashcpy(b->branch_tree.versions[0].sha1,
1763 b->branch_tree.versions[1].sha1);
1695 - size_dbuf(&new_data, 97 + msglen
1764 + size_dbuf(&new_data, 114 + msglen
1765 + merge_count * 49
1766 + (author
1767 ? strlen(author) + strlen(committer)
@@ -1708,11 +1777,9 @@ static void cmd_new_commit(void)
1777 free(merge_list);
1778 merge_list = next;
1779 }
1711 - if (author)
1712 - sp += sprintf(sp, "%s\n", author);
1713 - else
1714 - sp += sprintf(sp, "author %s\n", committer + 10);
1715 - sp += sprintf(sp, "%s\n\n", committer);
1780 + sp += sprintf(sp, "author %s\n", author ? author : committer);
1781 + sp += sprintf(sp, "committer %s\n", committer);
1782 + *sp++ = '\n';
1783 memcpy(sp, msg, msglen);
1784 sp += msglen;
1785 free(author);
@@ -1780,7 +1847,7 @@ static void cmd_new_tag(void)
1847 /* tagger ... */
1848 if (strncmp("tagger ", command_buf.buf, 7))
1849 die("Expected tagger command, got %s", command_buf.buf);
1783 - tagger = strdup(command_buf.buf);
1850 + tagger = parse_ident(command_buf.buf + 7);
1851
1852 /* tag payload/message */
1853 read_next_command();
@@ -1792,7 +1859,8 @@ static void cmd_new_tag(void)
1859 sp += sprintf(sp, "object %s\n", sha1_to_hex(sha1));
1860 sp += sprintf(sp, "type %s\n", type_names[OBJ_COMMIT]);
1861 sp += sprintf(sp, "tag %s\n", t->name);
1795 - sp += sprintf(sp, "%s\n\n", tagger);
1862 + sp += sprintf(sp, "tagger %s\n", tagger);
1863 + *sp++ = '\n';
1864 memcpy(sp, msg, msglen);
1865 sp += msglen;
1866 free(tagger);
@@ -1835,7 +1903,7 @@ static void cmd_checkpoint(void)
1903 }
1904
1905 static const char fast_import_usage[] =
1838 -"git-fast-import [--depth=n] [--active-branches=n] [--export-marks=marks.file] [--branch-log=log]";
1906 +"git-fast-import [--date-format=f] [--max-pack-size=n] [--depth=n] [--active-branches=n] [--export-marks=marks.file]";
1907
1908 int main(int argc, const char **argv)
1909 {
@@ -1849,6 +1917,17 @@ int main(int argc, const char **argv)
1917
1918 if (*a != '-' || !strcmp(a, "--"))
1919 break;
1920 + else if (!strncmp(a, "--date-format=", 14)) {
1921 + const char *fmt = a + 14;
1922 + if (!strcmp(fmt, "raw"))
1923 + whenspec = WHENSPEC_RAW;
1924 + else if (!strcmp(fmt, "rfc2822"))
1925 + whenspec = WHENSPEC_RFC2822;
1926 + else if (!strcmp(fmt, "now"))
1927 + whenspec = WHENSPEC_NOW;
1928 + else
1929 + die("unknown --date-format argument %s", fmt);
1930 + }
1931 else if (!strncmp(a, "--max-pack-size=", 16))
1932 max_packsize = strtoumax(a + 16, NULL, 0) * 1024 * 1024;
1933 else if (!strncmp(a, "--depth=", 8))
t/t9300-fast-import.sh
+36
@@ -240,4 +240,40 @@ test_expect_success \
240 'git-cat-file blob branch:newdir/exec.sh >actual &&
241 diff -u expect actual'
242
243 +###
244 +### series E
245 +###
246 +
247 +cat >input <<INPUT_END
248 +commit refs/heads/branch
249 +author $GIT_AUTHOR_NAME <$GIT_AUTHOR_EMAIL> Tue Feb 6 11:22:18 2007 -0500
250 +committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> Tue Feb 6 12:35:02 2007 -0500
251 +data <<COMMIT
252 +RFC 2822 type date
253 +COMMIT
254 +
255 +from refs/heads/branch^0
256 +
257 +INPUT_END
258 +test_expect_failure \
259 + 'E: rfc2822 date, --date-format=raw' \
260 + 'git-fast-import --date-format=raw <input'
261 +test_expect_success \
262 + 'E: rfc2822 date, --date-format=rfc2822' \
263 + 'git-fast-import --date-format=rfc2822 <input'
264 +test_expect_success \
265 + 'E: verify pack' \
266 + 'for p in .git/objects/pack/*.pack;do git-verify-pack $p||exit;done'
267 +
268 +cat >expect <<EOF
269 +author $GIT_AUTHOR_NAME <$GIT_AUTHOR_EMAIL> 1170778938 -0500
270 +committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1170783302 -0500
271 +
272 +RFC 2822 type date
273 +EOF
274 +test_expect_success \
275 + 'E: verify commit' \
276 + 'git-cat-file commit branch | sed 1,2d >actual &&
277 + diff -u expect actual'
278 +
279 test_done