752414ae43 · matheusbsbs/matheus-bagliano

@@ -0,0 +1,797 @@

1 +Git hash function transition

2 +============================

3 +

4 +Objective

5 +---------

6 +Migrate Git from SHA-1 to a stronger hash function.

7 +

8 +Background

9 +----------

10 +At its core, the Git version control system is a content addressable

11 +filesystem. It uses the SHA-1 hash function to name content. For

12 +example, files, directories, and revisions are referred to by hash

13 +values unlike in other traditional version control systems where files

14 +or versions are referred to via sequential numbers. The use of a hash

15 +function to address its content delivers a few advantages:

16 +

17 +* Integrity checking is easy. Bit flips, for example, are easily

18 + detected, as the hash of corrupted content does not match its name.

19 +* Lookup of objects is fast.

20 +

21 +Using a cryptographically secure hash function brings additional

22 +advantages:

23 +

24 +* Object names can be signed and third parties can trust the hash to

25 + address the signed object and all objects it references.

26 +* Communication using Git protocol and out of band communication

27 + methods have a short reliable string that can be used to reliably

28 + address stored content.

29 +

30 +Over time some flaws in SHA-1 have been discovered by security

31 +researchers. https://shattered.io demonstrated a practical SHA-1 hash

32 +collision. As a result, SHA-1 cannot be considered cryptographically

33 +secure any more. This impacts the communication of hash values because

34 +we cannot trust that a given hash value represents the known good

35 +version of content that the speaker intended.

36 +

37 +SHA-1 still possesses the other properties such as fast object lookup

38 +and safe error checking, but other hash functions are equally suitable

39 +that are believed to be cryptographically secure.

40 +

41 +Goals

42 +-----

43 +Where NewHash is a strong 256-bit hash function to replace SHA-1 (see

44 +"Selection of a New Hash", below):

45 +

46 +1. The transition to NewHash can be done one local repository at a time.

47 + a. Requiring no action by any other party.

48 + b. A NewHash repository can communicate with SHA-1 Git servers

49 + (push/fetch).

50 + c. Users can use SHA-1 and NewHash identifiers for objects

51 + interchangeably (see "Object names on the command line", below).

52 + d. New signed objects make use of a stronger hash function than

53 + SHA-1 for their security guarantees.

54 +2. Allow a complete transition away from SHA-1.

55 + a. Local metadata for SHA-1 compatibility can be removed from a

56 + repository if compatibility with SHA-1 is no longer needed.

57 +3. Maintainability throughout the process.

58 + a. The object format is kept simple and consistent.

59 + b. Creation of a generalized repository conversion tool.

60 +

61 +Non-Goals

62 +---------

63 +1. Add NewHash support to Git protocol. This is valuable and the

64 + logical next step but it is out of scope for this initial design.

65 +2. Transparently improving the security of existing SHA-1 signed

66 + objects.

67 +3. Intermixing objects using multiple hash functions in a single

68 + repository.

69 +4. Taking the opportunity to fix other bugs in Git's formats and

70 + protocols.

71 +5. Shallow clones and fetches into a NewHash repository. (This will

72 + change when we add NewHash support to Git protocol.)

73 +6. Skip fetching some submodules of a project into a NewHash

74 + repository. (This also depends on NewHash support in Git

75 + protocol.)

76 +

77 +Overview

78 +--------

79 +We introduce a new repository format extension. Repositories with this

80 +extension enabled use NewHash instead of SHA-1 to name their objects.

81 +This affects both object names and object content --- both the names

82 +of objects and all references to other objects within an object are

83 +switched to the new hash function.

84 +

85 +NewHash repositories cannot be read by older versions of Git.

86 +

87 +Alongside the packfile, a NewHash repository stores a bidirectional

88 +mapping between NewHash and SHA-1 object names. The mapping is generated

89 +locally and can be verified using "git fsck". Object lookups use this

90 +mapping to allow naming objects using either their SHA-1 and NewHash names

91 +interchangeably.

92 +

93 +"git cat-file" and "git hash-object" gain options to display an object

94 +in its sha1 form and write an object given its sha1 form. This

95 +requires all objects referenced by that object to be present in the

96 +object database so that they can be named using the appropriate name

97 +(using the bidirectional hash mapping).

98 +

99 +Fetches from a SHA-1 based server convert the fetched objects into

100 +NewHash form and record the mapping in the bidirectional mapping table

101 +(see below for details). Pushes to a SHA-1 based server convert the

102 +objects being pushed into sha1 form so the server does not have to be

103 +aware of the hash function the client is using.

104 +

105 +Detailed Design

106 +---------------

107 +Repository format extension

108 +~~~~~~~~~~~~~~~~~~~~~~~~~~~

109 +A NewHash repository uses repository format version `1` (see

110 +Documentation/technical/repository-version.txt) with extensions

111 +`objectFormat` and `compatObjectFormat`:

112 +

113 + [core]

114 + repositoryFormatVersion = 1

115 + [extensions]

116 + objectFormat = newhash

117 + compatObjectFormat = sha1

118 +

119 +Specifying a repository format extension ensures that versions of Git

120 +not aware of NewHash do not try to operate on these repositories,

121 +instead producing an error message:

122 +

123 + $ git status

124 + fatal: unknown repository extensions found:

125 + objectformat

126 + compatobjectformat

127 +

128 +See the "Transition plan" section below for more details on these

129 +repository extensions.

130 +

131 +Object names

132 +~~~~~~~~~~~~

133 +Objects can be named by their 40 hexadecimal digit sha1-name or 64

134 +hexadecimal digit newhash-name, plus names derived from those (see

135 +gitrevisions(7)).

136 +

137 +The sha1-name of an object is the SHA-1 of the concatenation of its

138 +type, length, a nul byte, and the object's sha1-content. This is the

139 +traditional <sha1> used in Git to name objects.

140 +

141 +The newhash-name of an object is the NewHash of the concatenation of its

142 +type, length, a nul byte, and the object's newhash-content.

143 +

144 +Object format

145 +~~~~~~~~~~~~~

146 +The content as a byte sequence of a tag, commit, or tree object named

147 +by sha1 and newhash differ because an object named by newhash-name refers to

148 +other objects by their newhash-names and an object named by sha1-name

149 +refers to other objects by their sha1-names.

150 +

151 +The newhash-content of an object is the same as its sha1-content, except

152 +that objects referenced by the object are named using their newhash-names

153 +instead of sha1-names. Because a blob object does not refer to any

154 +other object, its sha1-content and newhash-content are the same.

155 +

156 +The format allows round-trip conversion between newhash-content and

157 +sha1-content.

158 +

159 +Object storage

160 +~~~~~~~~~~~~~~

161 +Loose objects use zlib compression and packed objects use the packed

162 +format described in Documentation/technical/pack-format.txt, just like

163 +today. The content that is compressed and stored uses newhash-content

164 +instead of sha1-content.

165 +

166 +Pack index

167 +~~~~~~~~~~

168 +Pack index (.idx) files use a new v3 format that supports multiple

169 +hash functions. They have the following format (all integers are in

170 +network byte order):

171 +

172 +- A header appears at the beginning and consists of the following:

173 + - The 4-byte pack index signature: '\377t0c'

174 + - 4-byte version number: 3

175 + - 4-byte length of the header section, including the signature and

176 + version number

177 + - 4-byte number of objects contained in the pack

178 + - 4-byte number of object formats in this pack index: 2

179 + - For each object format:

180 + - 4-byte format identifier (e.g., 'sha1' for SHA-1)

181 + - 4-byte length in bytes of shortened object names. This is the

182 + shortest possible length needed to make names in the shortened

183 + object name table unambiguous.

184 + - 4-byte integer, recording where tables relating to this format

185 + are stored in this index file, as an offset from the beginning.

186 + - 4-byte offset to the trailer from the beginning of this file.

187 + - Zero or more additional key/value pairs (4-byte key, 4-byte

188 + value). Only one key is supported: 'PSRC'. See the "Loose objects

189 + and unreachable objects" section for supported values and how this

190 + is used. All other keys are reserved. Readers must ignore

191 + unrecognized keys.

192 +- Zero or more NUL bytes. This can optionally be used to improve the

193 + alignment of the full object name table below.

194 +- Tables for the first object format:

195 + - A sorted table of shortened object names. These are prefixes of

196 + the names of all objects in this pack file, packed together

197 + without offset values to reduce the cache footprint of the binary

198 + search for a specific object name.

199 +

200 + - A table of full object names in pack order. This allows resolving

201 + a reference to "the nth object in the pack file" (from a

202 + reachability bitmap or from the next table of another object

203 + format) to its object name.

204 +

205 + - A table of 4-byte values mapping object name order to pack order.

206 + For an object in the table of sorted shortened object names, the

207 + value at the corresponding index in this table is the index in the

208 + previous table for that same object.

209 +

210 + This can be used to look up the object in reachability bitmaps or

211 + to look up its name in another object format.

212 +

213 + - A table of 4-byte CRC32 values of the packed object data, in the

214 + order that the objects appear in the pack file. This is to allow

215 + compressed data to be copied directly from pack to pack during

216 + repacking without undetected data corruption.

217 +

218 + - A table of 4-byte offset values. For an object in the table of

219 + sorted shortened object names, the value at the corresponding

220 + index in this table indicates where that object can be found in

221 + the pack file. These are usually 31-bit pack file offsets, but

222 + large offsets are encoded as an index into the next table with the

223 + most significant bit set.

224 +

225 + - A table of 8-byte offset entries (empty for pack files less than

226 + 2 GiB). Pack files are organized with heavily used objects toward

227 + the front, so most object references should not need to refer to

228 + this table.

229 +- Zero or more NUL bytes.

230 +- Tables for the second object format, with the same layout as above,

231 + up to and not including the table of CRC32 values.

232 +- Zero or more NUL bytes.

233 +- The trailer consists of the following:

234 + - A copy of the 20-byte NewHash checksum at the end of the

235 + corresponding packfile.

236 +

237 + - 20-byte NewHash checksum of all of the above.

238 +

239 +Loose object index

240 +~~~~~~~~~~~~~~~~~~

241 +A new file $GIT_OBJECT_DIR/loose-object-idx contains information about

242 +all loose objects. Its format is

243 +

244 + # loose-object-idx

245 + (newhash-name SP sha1-name LF)*

246 +

247 +where the object names are in hexadecimal format. The file is not

248 +sorted.

249 +

250 +The loose object index is protected against concurrent writes by a

251 +lock file $GIT_OBJECT_DIR/loose-object-idx.lock. To add a new loose

252 +object:

253 +

254 +1. Write the loose object to a temporary file, like today.

255 +2. Open loose-object-idx.lock with O_CREAT | O_EXCL to acquire the lock.

256 +3. Rename the loose object into place.

257 +4. Open loose-object-idx with O_APPEND and write the new object

258 +5. Unlink loose-object-idx.lock to release the lock.

259 +

260 +To remove entries (e.g. in "git pack-refs" or "git-prune"):

261 +

262 +1. Open loose-object-idx.lock with O_CREAT | O_EXCL to acquire the

263 + lock.

264 +2. Write the new content to loose-object-idx.lock.

265 +3. Unlink any loose objects being removed.

266 +4. Rename to replace loose-object-idx, releasing the lock.

267 +

268 +Translation table

269 +~~~~~~~~~~~~~~~~~

270 +The index files support a bidirectional mapping between sha1-names

271 +and newhash-names. The lookup proceeds similarly to ordinary object

272 +lookups. For example, to convert a sha1-name to a newhash-name:

273 +

274 + 1. Look for the object in idx files. If a match is present in the

275 + idx's sorted list of truncated sha1-names, then:

276 + a. Read the corresponding entry in the sha1-name order to pack

277 + name order mapping.

278 + b. Read the corresponding entry in the full sha1-name table to

279 + verify we found the right object. If it is, then

280 + c. Read the corresponding entry in the full newhash-name table.

281 + That is the object's newhash-name.

282 + 2. Check for a loose object. Read lines from loose-object-idx until

283 + we find a match.

284 +

285 +Step (1) takes the same amount of time as an ordinary object lookup:

286 +O(number of packs * log(objects per pack)). Step (2) takes O(number of

287 +loose objects) time. To maintain good performance it will be necessary

288 +to keep the number of loose objects low. See the "Loose objects and

289 +unreachable objects" section below for more details.

290 +

291 +Since all operations that make new objects (e.g., "git commit") add

292 +the new objects to the corresponding index, this mapping is possible

293 +for all objects in the object store.

294 +

295 +Reading an object's sha1-content

296 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

297 +The sha1-content of an object can be read by converting all newhash-names

298 +its newhash-content references to sha1-names using the translation table.

299 +

300 +Fetch

301 +~~~~~

302 +Fetching from a SHA-1 based server requires translating between SHA-1

303 +and NewHash based representations on the fly.

304 +

305 +SHA-1s named in the ref advertisement that are present on the client

306 +can be translated to NewHash and looked up as local objects using the

307 +translation table.

308 +

309 +Negotiation proceeds as today. Any "have"s generated locally are

310 +converted to SHA-1 before being sent to the server, and SHA-1s

311 +mentioned by the server are converted to NewHash when looking them up

312 +locally.

313 +

314 +After negotiation, the server sends a packfile containing the

315 +requested objects. We convert the packfile to NewHash format using

316 +the following steps:

317 +

318 +1. index-pack: inflate each object in the packfile and compute its

319 + SHA-1. Objects can contain deltas in OBJ_REF_DELTA format against

320 + objects the client has locally. These objects can be looked up

321 + using the translation table and their sha1-content read as

322 + described above to resolve the deltas.

323 +2. topological sort: starting at the "want"s from the negotiation

324 + phase, walk through objects in the pack and emit a list of them,

325 + excluding blobs, in reverse topologically sorted order, with each

326 + object coming later in the list than all objects it references.

327 + (This list only contains objects reachable from the "wants". If the

328 + pack from the server contained additional extraneous objects, then

329 + they will be discarded.)

330 +3. convert to newhash: open a new (newhash) packfile. Read the topologically

331 + sorted list just generated. For each object, inflate its

332 + sha1-content, convert to newhash-content, and write it to the newhash

333 + pack. Record the new sha1<->newhash mapping entry for use in the idx.

334 +4. sort: reorder entries in the new pack to match the order of objects

335 + in the pack the server generated and include blobs. Write a newhash idx

336 + file

337 +5. clean up: remove the SHA-1 based pack file, index, and

338 + topologically sorted list obtained from the server in steps 1

339 + and 2.

340 +

341 +Step 3 requires every object referenced by the new object to be in the

342 +translation table. This is why the topological sort step is necessary.

343 +

344 +As an optimization, step 1 could write a file describing what non-blob

345 +objects each object it has inflated from the packfile references. This

346 +makes the topological sort in step 2 possible without inflating the

347 +objects in the packfile for a second time. The objects need to be

348 +inflated again in step 3, for a total of two inflations.

349 +

350 +Step 4 is probably necessary for good read-time performance. "git

351 +pack-objects" on the server optimizes the pack file for good data

352 +locality (see Documentation/technical/pack-heuristics.txt).

353 +

354 +Details of this process are likely to change. It will take some

355 +experimenting to get this to perform well.

356 +

357 +Push

358 +~~~~

359 +Push is simpler than fetch because the objects referenced by the

360 +pushed objects are already in the translation table. The sha1-content

361 +of each object being pushed can be read as described in the "Reading

362 +an object's sha1-content" section to generate the pack written by git

363 +send-pack.

364 +

365 +Signed Commits

366 +~~~~~~~~~~~~~~

367 +We add a new field "gpgsig-newhash" to the commit object format to allow

368 +signing commits without relying on SHA-1. It is similar to the

369 +existing "gpgsig" field. Its signed payload is the newhash-content of the

370 +commit object with any "gpgsig" and "gpgsig-newhash" fields removed.

371 +

372 +This means commits can be signed

373 +1. using SHA-1 only, as in existing signed commit objects

374 +2. using both SHA-1 and NewHash, by using both gpgsig-newhash and gpgsig

375 + fields.

376 +3. using only NewHash, by only using the gpgsig-newhash field.

377 +

378 +Old versions of "git verify-commit" can verify the gpgsig signature in

379 +cases (1) and (2) without modifications and view case (3) as an

380 +ordinary unsigned commit.

381 +

382 +Signed Tags

383 +~~~~~~~~~~~

384 +We add a new field "gpgsig-newhash" to the tag object format to allow

385 +signing tags without relying on SHA-1. Its signed payload is the

386 +newhash-content of the tag with its gpgsig-newhash field and "-----BEGIN PGP

387 +SIGNATURE-----" delimited in-body signature removed.

388 +

389 +This means tags can be signed

390 +1. using SHA-1 only, as in existing signed tag objects

391 +2. using both SHA-1 and NewHash, by using gpgsig-newhash and an in-body

392 + signature.

393 +3. using only NewHash, by only using the gpgsig-newhash field.

394 +

395 +Mergetag embedding

396 +~~~~~~~~~~~~~~~~~~

397 +The mergetag field in the sha1-content of a commit contains the

398 +sha1-content of a tag that was merged by that commit.

399 +

400 +The mergetag field in the newhash-content of the same commit contains the

401 +newhash-content of the same tag.

402 +

403 +Submodules

404 +~~~~~~~~~~

405 +To convert recorded submodule pointers, you need to have the converted

406 +submodule repository in place. The translation table of the submodule

407 +can be used to look up the new hash.

408 +

409 +Loose objects and unreachable objects

410 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

411 +Fast lookups in the loose-object-idx require that the number of loose

412 +objects not grow too high.

413 +

414 +"git gc --auto" currently waits for there to be 6700 loose objects

415 +present before consolidating them into a packfile. We will need to

416 +measure to find a more appropriate threshold for it to use.

417 +

418 +"git gc --auto" currently waits for there to be 50 packs present

419 +before combining packfiles. Packing loose objects more aggressively

420 +may cause the number of pack files to grow too quickly. This can be

421 +mitigated by using a strategy similar to Martin Fick's exponential

422 +rolling garbage collection script:

423 +https://gerrit-review.googlesource.com/c/gerrit/+/35215

424 +

425 +"git gc" currently expels any unreachable objects it encounters in

426 +pack files to loose objects in an attempt to prevent a race when

427 +pruning them (in case another process is simultaneously writing a new

428 +object that refers to the about-to-be-deleted object). This leads to

429 +an explosion in the number of loose objects present and disk space

430 +usage due to the objects in delta form being replaced with independent

431 +loose objects. Worse, the race is still present for loose objects.

432 +

433 +Instead, "git gc" will need to move unreachable objects to a new

434 +packfile marked as UNREACHABLE_GARBAGE (using the PSRC field; see

435 +below). To avoid the race when writing new objects referring to an

436 +about-to-be-deleted object, code paths that write new objects will

437 +need to copy any objects from UNREACHABLE_GARBAGE packs that they

438 +refer to to new, non-UNREACHABLE_GARBAGE packs (or loose objects).

439 +UNREACHABLE_GARBAGE are then safe to delete if their creation time (as

440 +indicated by the file's mtime) is long enough ago.

441 +

442 +To avoid a proliferation of UNREACHABLE_GARBAGE packs, they can be

443 +combined under certain circumstances. If "gc.garbageTtl" is set to

444 +greater than one day, then packs created within a single calendar day,

445 +UTC, can be coalesced together. The resulting packfile would have an

446 +mtime before midnight on that day, so this makes the effective maximum

447 +ttl the garbageTtl + 1 day. If "gc.garbageTtl" is less than one day,

448 +then we divide the calendar day into intervals one-third of that ttl

449 +in duration. Packs created within the same interval can be coalesced

450 +together. The resulting packfile would have an mtime before the end of

451 +the interval, so this makes the effective maximum ttl equal to the

452 +garbageTtl * 4/3.

453 +

454 +This rule comes from Thirumala Reddy Mutchukota's JGit change

455 +https://git.eclipse.org/r/90465.

456 +

457 +The UNREACHABLE_GARBAGE setting goes in the PSRC field of the pack

458 +index. More generally, that field indicates where a pack came from:

459 +

460 + - 1 (PACK_SOURCE_RECEIVE) for a pack received over the network

461 + - 2 (PACK_SOURCE_AUTO) for a pack created by a lightweight

462 + "gc --auto" operation

463 + - 3 (PACK_SOURCE_GC) for a pack created by a full gc

464 + - 4 (PACK_SOURCE_UNREACHABLE_GARBAGE) for potential garbage

465 + discovered by gc

466 + - 5 (PACK_SOURCE_INSERT) for locally created objects that were

467 + written directly to a pack file, e.g. from "git add ."

468 +

469 +This information can be useful for debugging and for "gc --auto" to

470 +make appropriate choices about which packs to coalesce.

471 +

472 +Caveats

473 +-------

474 +Invalid objects

475 +~~~~~~~~~~~~~~~

476 +The conversion from sha1-content to newhash-content retains any

477 +brokenness in the original object (e.g., tree entry modes encoded with

478 +leading 0, tree objects whose paths are not sorted correctly, and

479 +commit objects without an author or committer). This is a deliberate

480 +feature of the design to allow the conversion to round-trip.

481 +

482 +More profoundly broken objects (e.g., a commit with a truncated "tree"

483 +header line) cannot be converted but were not usable by current Git

484 +anyway.

485 +

486 +Shallow clone and submodules

487 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~

488 +Because it requires all referenced objects to be available in the

489 +locally generated translation table, this design does not support

490 +shallow clone or unfetched submodules. Protocol improvements might

491 +allow lifting this restriction.

492 +

493 +Alternates

494 +~~~~~~~~~~

495 +For the same reason, a newhash repository cannot borrow objects from a

496 +sha1 repository using objects/info/alternates or

497 +$GIT_ALTERNATE_OBJECT_REPOSITORIES.

498 +

499 +git notes

500 +~~~~~~~~~

501 +The "git notes" tool annotates objects using their sha1-name as key.

502 +This design does not describe a way to migrate notes trees to use

503 +newhash-names. That migration is expected to happen separately (for

504 +example using a file at the root of the notes tree to describe which

505 +hash it uses).

506 +

507 +Server-side cost

508 +~~~~~~~~~~~~~~~~

509 +Until Git protocol gains NewHash support, using NewHash based storage

510 +on public-facing Git servers is strongly discouraged. Once Git

511 +protocol gains NewHash support, NewHash based servers are likely not

512 +to support SHA-1 compatibility, to avoid what may be a very expensive

513 +hash reencode during clone and to encourage peers to modernize.

514 +

515 +The design described here allows fetches by SHA-1 clients of a

516 +personal NewHash repository because it's not much more difficult than

517 +allowing pushes from that repository. This support needs to be guarded

518 +by a configuration option --- servers like git.kernel.org that serve a

519 +large number of clients would not be expected to bear that cost.

520 +

521 +Meaning of signatures

522 +~~~~~~~~~~~~~~~~~~~~~

523 +The signed payload for signed commits and tags does not explicitly

524 +name the hash used to identify objects. If some day Git adopts a new

525 +hash function with the same length as the current SHA-1 (40

526 +hexadecimal digit) or NewHash (64 hexadecimal digit) objects then the

527 +intent behind the PGP signed payload in an object signature is

528 +unclear:

529 +

530 + object e7e07d5a4fcc2a203d9873968ad3e6bd4d7419d7

531 + type commit

532 + tag v2.12.0

533 + tagger Junio C Hamano <gitster@pobox.com> 1487962205 -0800

534 +

535 + Git 2.12

536 +

537 +Does this mean Git v2.12.0 is the commit with sha1-name

538 +e7e07d5a4fcc2a203d9873968ad3e6bd4d7419d7 or the commit with

539 +new-40-digit-hash-name e7e07d5a4fcc2a203d9873968ad3e6bd4d7419d7?

540 +

541 +Fortunately NewHash and SHA-1 have different lengths. If Git starts

542 +using another hash with the same length to name objects, then it will

543 +need to change the format of signed payloads using that hash to

544 +address this issue.

545 +

546 +Object names on the command line

547 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

548 +To support the transition (see Transition plan below), this design

549 +supports four different modes of operation:

550 +

551 + 1. ("dark launch") Treat object names input by the user as SHA-1 and

552 + convert any object names written to output to SHA-1, but store

553 + objects using NewHash. This allows users to test the code with no

554 + visible behavior change except for performance. This allows

555 + allows running even tests that assume the SHA-1 hash function, to

556 + sanity-check the behavior of the new mode.

557 +

558 + 2. ("early transition") Allow both SHA-1 and NewHash object names in

559 + input. Any object names written to output use SHA-1. This allows

560 + users to continue to make use of SHA-1 to communicate with peers

561 + (e.g. by email) that have not migrated yet and prepares for mode 3.

562 +

563 + 3. ("late transition") Allow both SHA-1 and NewHash object names in

564 + input. Any object names written to output use NewHash. In this

565 + mode, users are using a more secure object naming method by

566 + default. The disruption is minimal as long as most of their peers

567 + are in mode 2 or mode 3.

568 +

569 + 4. ("post-transition") Treat object names input by the user as

570 + NewHash and write output using NewHash. This is safer than mode 3

571 + because there is less risk that input is incorrectly interpreted

572 + using the wrong hash function.

573 +

574 +The mode is specified in configuration.

575 +

576 +The user can also explicitly specify which format to use for a

577 +particular revision specifier and for output, overriding the mode. For

578 +example:

579 +

580 +git --output-format=sha1 log abac87a^{sha1}..f787cac^{newhash}

581 +

582 +Selection of a New Hash

583 +-----------------------

584 +In early 2005, around the time that Git was written, Xiaoyun Wang,

585 +Yiqun Lisa Yin, and Hongbo Yu announced an attack finding SHA-1

586 +collisions in 2^69 operations. In August they published details.

587 +Luckily, no practical demonstrations of a collision in full SHA-1 were

588 +published until 10 years later, in 2017.

589 +

590 +The hash function NewHash to replace SHA-1 should be stronger than

591 +SHA-1 was: we would like it to be trustworthy and useful in practice

592 +for at least 10 years.

593 +

594 +Some other relevant properties:

595 +

596 +1. A 256-bit hash (long enough to match common security practice; not

597 + excessively long to hurt performance and disk usage).

598 +

599 +2. High quality implementations should be widely available (e.g. in

600 + OpenSSL).

601 +

602 +3. The hash function's properties should match Git's needs (e.g. Git

603 + requires collision and 2nd preimage resistance and does not require

604 + length extension resistance).

605 +

606 +4. As a tiebreaker, the hash should be fast to compute (fortunately

607 + many contenders are faster than SHA-1).

608 +

609 +Some hashes under consideration are SHA-256, SHA-512/256, SHA-256x16,

610 +K12, and BLAKE2bp-256.

611 +

612 +Transition plan

613 +---------------

614 +Some initial steps can be implemented independently of one another:

615 +- adding a hash function API (vtable)

616 +- teaching fsck to tolerate the gpgsig-newhash field

617 +- excluding gpgsig-* from the fields copied by "git commit --amend"

618 +- annotating tests that depend on SHA-1 values with a SHA1 test

619 + prerequisite

620 +- using "struct object_id", GIT_MAX_RAWSZ, and GIT_MAX_HEXSZ

621 + consistently instead of "unsigned char *" and the hardcoded

622 + constants 20 and 40.

623 +- introducing index v3

624 +- adding support for the PSRC field and safer object pruning

625 +

626 +

627 +The first user-visible change is the introduction of the objectFormat

628 +extension (without compatObjectFormat). This requires:

629 +- implementing the loose-object-idx

630 +- teaching fsck about this mode of operation

631 +- using the hash function API (vtable) when computing object names

632 +- signing objects and verifying signatures

633 +- rejecting attempts to fetch from or push to an incompatible

634 + repository

635 +

636 +Next comes introduction of compatObjectFormat:

637 +- translating object names between object formats

638 +- translating object content between object formats

639 +- generating and verifying signatures in the compat format

640 +- adding appropriate index entries when adding a new object to the

641 + object store

642 +- --output-format option

643 +- ^{sha1} and ^{newhash} revision notation

644 +- configuration to specify default input and output format (see

645 + "Object names on the command line" above)

646 +

647 +The next step is supporting fetches and pushes to SHA-1 repositories:

648 +- allow pushes to a repository using the compat format

649 +- generate a topologically sorted list of the SHA-1 names of fetched

650 + objects

651 +- convert the fetched packfile to newhash format and generate an idx

652 + file

653 +- re-sort to match the order of objects in the fetched packfile

654 +

655 +The infrastructure supporting fetch also allows converting an existing

656 +repository. In converted repositories and new clones, end users can

657 +gain support for the new hash function without any visible change in

658 +behavior (see "dark launch" in the "Object names on the command line"

659 +section). In particular this allows users to verify NewHash signatures

660 +on objects in the repository, and it should ensure the transition code

661 +is stable in production in preparation for using it more widely.

662 +

663 +Over time projects would encourage their users to adopt the "early

664 +transition" and then "late transition" modes to take advantage of the

665 +new, more futureproof NewHash object names.

666 +

667 +When objectFormat and compatObjectFormat are both set, commands

668 +generating signatures would generate both SHA-1 and NewHash signatures

669 +by default to support both new and old users.

670 +

671 +In projects using NewHash heavily, users could be encouraged to adopt

672 +the "post-transition" mode to avoid accidentally making implicit use

673 +of SHA-1 object names.

674 +

675 +Once a critical mass of users have upgraded to a version of Git that

676 +can verify NewHash signatures and have converted their existing

677 +repositories to support verifying them, we can add support for a

678 +setting to generate only NewHash signatures. This is expected to be at

679 +least a year later.

680 +

681 +That is also a good moment to advertise the ability to convert

682 +repositories to use NewHash only, stripping out all SHA-1 related

683 +metadata. This improves performance by eliminating translation

684 +overhead and security by avoiding the possibility of accidentally

685 +relying on the safety of SHA-1.

686 +

687 +Updating Git's protocols to allow a server to specify which hash

688 +functions it supports is also an important part of this transition. It

689 +is not discussed in detail in this document but this transition plan

690 +assumes it happens. :)

691 +

692 +Alternatives considered

693 +-----------------------

694 +Upgrading everyone working on a particular project on a flag day

695 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

696 +Projects like the Linux kernel are large and complex enough that

697 +flipping the switch for all projects based on the repository at once

698 +is infeasible.

699 +

700 +Not only would all developers and server operators supporting

701 +developers have to switch on the same flag day, but supporting tooling

702 +(continuous integration, code review, bug trackers, etc) would have to

703 +be adapted as well. This also makes it difficult to get early feedback

704 +from some project participants testing before it is time for mass

705 +adoption.

706 +

707 +Using hash functions in parallel

708 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

709 +(e.g. https://public-inbox.org/git/22708.8913.864049.452252@chiark.greenend.org.uk/ )

710 +Objects newly created would be addressed by the new hash, but inside

711 +such an object (e.g. commit) it is still possible to address objects

712 +using the old hash function.

713 +* You cannot trust its history (needed for bisectability) in the

714 + future without further work

715 +* Maintenance burden as the number of supported hash functions grows

716 + (they will never go away, so they accumulate). In this proposal, by

717 + comparison, converted objects lose all references to SHA-1.

718 +

719 +Signed objects with multiple hashes

720 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

721 +Instead of introducing the gpgsig-newhash field in commit and tag objects

722 +for newhash-content based signatures, an earlier version of this design

723 +added "hash newhash <newhash-name>" fields to strengthen the existing

724 +sha1-content based signatures.

725 +

726 +In other words, a single signature was used to attest to the object

727 +content using both hash functions. This had some advantages:

728 +* Using one signature instead of two speeds up the signing process.

729 +* Having one signed payload with both hashes allows the signer to

730 + attest to the sha1-name and newhash-name referring to the same object.

731 +* All users consume the same signature. Broken signatures are likely

732 + to be detected quickly using current versions of git.

733 +

734 +However, it also came with disadvantages:

735 +* Verifying a signed object requires access to the sha1-names of all

736 + objects it references, even after the transition is complete and

737 + translation table is no longer needed for anything else. To support

738 + this, the design added fields such as "hash sha1 tree <sha1-name>"

739 + and "hash sha1 parent <sha1-name>" to the newhash-content of a signed

740 + commit, complicating the conversion process.

741 +* Allowing signed objects without a sha1 (for after the transition is

742 + complete) complicated the design further, requiring a "nohash sha1"

743 + field to suppress including "hash sha1" fields in the newhash-content

744 + and signed payload.

745 +

746 +Lazily populated translation table

747 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

748 +Some of the work of building the translation table could be deferred to

749 +push time, but that would significantly complicate and slow down pushes.

750 +Calculating the sha1-name at object creation time at the same time it is

751 +being streamed to disk and having its newhash-name calculated should be

752 +an acceptable cost.

753 +

754 +Document History

755 +----------------

756 +

757 +2017-03-03

758 +bmwill@google.com, jonathantanmy@google.com, jrnieder@gmail.com,

759 +sbeller@google.com

760 +

761 +Initial version sent to

762 +http://public-inbox.org/git/20170304011251.GA26789@aiede.mtv.corp.google.com

763 +

764 +2017-03-03 jrnieder@gmail.com

765 +Incorporated suggestions from jonathantanmy and sbeller:

766 +* describe purpose of signed objects with each hash type

767 +* redefine signed object verification using object content under the

768 + first hash function

769 +

770 +2017-03-06 jrnieder@gmail.com

771 +* Use SHA3-256 instead of SHA2 (thanks, Linus and brian m. carlson).[1][2]

772 +* Make sha3-based signatures a separate field, avoiding the need for

773 + "hash" and "nohash" fields (thanks to peff[3]).

774 +* Add a sorting phase to fetch (thanks to Junio for noticing the need

775 + for this).

776 +* Omit blobs from the topological sort during fetch (thanks to peff).

777 +* Discuss alternates, git notes, and git servers in the caveats

778 + section (thanks to Junio Hamano, brian m. carlson[4], and Shawn

779 + Pearce).

780 +* Clarify language throughout (thanks to various commenters,

781 + especially Junio).

782 +

783 +2017-09-27 jrnieder@gmail.com, sbeller@google.com

784 +* use placeholder NewHash instead of SHA3-256

785 +* describe criteria for picking a hash function.

786 +* include a transition plan (thanks especially to Brandon Williams

787 + for fleshing these ideas out)

788 +* define the translation table (thanks, Shawn Pearce[5], Jonathan

789 + Tan, and Masaya Suzuki)

790 +* avoid loose object overhead by packing more aggressively in

791 + "git gc --auto"

792 +

793 +[1] http://public-inbox.org/git/CA+55aFzJtejiCjV0e43+9oR3QuJK2PiFiLQemytoLpyJWe6P9w@mail.gmail.com/

794 +[2] http://public-inbox.org/git/CA+55aFz+gkAsDZ24zmePQuEs1XPS9BP_s8O7Q4wQ7LV7X5-oDA@mail.gmail.com/

795 +[3] http://public-inbox.org/git/20170306084353.nrns455dvkdsfgo5@sigill.intra.peff.net/

796 +[4] http://public-inbox.org/git/20170304224936.rqqtkdvfjgyezsht@genre.crustytoothpaste.net

797 +[5] https://public-inbox.org/git/CAJo=hJtoX9=AyLHHpUJS7fueV9ciZ_MNpnEPHUz8Whui6g9F0A@mail.gmail.com/

technical doc: add a design doc for hash function transition