2da14fad8f · matheusbsbs/matheus-bagliano

@@ -0,0 +1,468 @@

1 +Bundle URIs

2 +===========

3 +

4 +Git bundles are files that store a pack-file along with some extra metadata,

5 +including a set of refs and a (possibly empty) set of necessary commits. See

6 +linkgit:git-bundle[1] and link:bundle-format.txt[the bundle format] for more

7 +information.

8 +

9 +Bundle URIs are locations where Git can download one or more bundles in

10 +order to bootstrap the object database in advance of fetching the remaining

11 +objects from a remote.

12 +

13 +One goal is to speed up clones and fetches for users with poor network

14 +connectivity to the origin server. Another benefit is to allow heavy users,

15 +such as CI build farms, to use local resources for the majority of Git data

16 +and thereby reducing the load on the origin server.

17 +

18 +To enable the bundle URI feature, users can specify a bundle URI using

19 +command-line options or the origin server can advertise one or more URIs

20 +via a protocol v2 capability.

21 +

22 +Design Goals

23 +------------

24 +

25 +The bundle URI standard aims to be flexible enough to satisfy multiple

26 +workloads. The bundle provider and the Git client have several choices in

27 +how they create and consume bundle URIs.

28 +

29 +* Bundles can have whatever name the server desires. This name could refer

30 + to immutable data by using a hash of the bundle contents. However, this

31 + means that a new URI will be needed after every update of the content.

32 + This might be acceptable if the server is advertising the URI (and the

33 + server is aware of new bundles being generated) but would not be

34 + ergonomic for users using the command line option.

35 +

36 +* The bundles could be organized specifically for bootstrapping full

37 + clones, but could also be organized with the intention of bootstrapping

38 + incremental fetches. The bundle provider must decide on one of several

39 + organization schemes to minimize client downloads during incremental

40 + fetches, but the Git client can also choose whether to use bundles for

41 + either of these operations.

42 +

43 +* The bundle provider can choose to support full clones, partial clones,

44 + or both. The client can detect which bundles are appropriate for the

45 + repository's partial clone filter, if any.

46 +

47 +* The bundle provider can use a single bundle (for clones only), or a

48 + list of bundles. When using a list of bundles, the provider can specify

49 + whether or not the client needs _all_ of the bundle URIs for a full

50 + clone, or if _any_ one of the bundle URIs is sufficient. This allows the

51 + bundle provider to use different URIs for different geographies.

52 +

53 +* The bundle provider can organize the bundles using heuristics, such as

54 + creation tokens, to help the client prevent downloading bundles it does

55 + not need. When the bundle provider does not provide these heuristics,

56 + the client can use optimizations to minimize how much of the data is

57 + downloaded.

58 +

59 +* The bundle provider does not need to be associated with the Git server.

60 + The client can choose to use the bundle provider without it being

61 + advertised by the Git server.

62 +

63 +* The client can choose to discover bundle providers that are advertised

64 + by the Git server. This could happen during `git clone`, during

65 + `git fetch`, both, or neither. The user can choose which combination

66 + works best for them.

67 +

68 +* The client can choose to configure a bundle provider manually at any

69 + time. The client can also choose to specify a bundle provider manually

70 + as a command-line option to `git clone`.

71 +

72 +Each repository is different and every Git server has different needs.

73 +Hopefully the bundle URI feature is flexible enough to satisfy all needs.

74 +If not, then the feature can be extended through its versioning mechanism.

75 +

76 +Server requirements

77 +-------------------

78 +

79 +To provide a server-side implementation of bundle servers, no other parts

80 +of the Git protocol are required. This allows server maintainers to use

81 +static content solutions such as CDNs in order to serve the bundle files.

82 +

83 +At the current scope of the bundle URI feature, all URIs are expected to

84 +be HTTP(S) URLs where content is downloaded to a local file using a `GET`

85 +request to that URL. The server could include authentication requirements

86 +to those requests with the aim of triggering the configured credential

87 +helper for secure access. (Future extensions could use "file://" URIs or

88 +SSH URIs.)

89 +

90 +Assuming a `200 OK` response from the server, the content at the URL is

91 +inspected. First, Git attempts to parse the file as a bundle file of

92 +version 2 or higher. If the file is not a bundle, then the file is parsed

93 +as a plain-text file using Git's config parser. The key-value pairs in

94 +that config file are expected to describe a list of bundle URIs. If

95 +neither of these parse attempts succeed, then Git will report an error to

96 +the user that the bundle URI provided erroneous data.

97 +

98 +Any other data provided by the server is considered erroneous.

99 +

100 +Bundle Lists

101 +------------

102 +

103 +The Git server can advertise bundle URIs using a set of `key=value` pairs.

104 +A bundle URI can also serve a plain-text file in the Git config format

105 +containing these same `key=value` pairs. In both cases, we consider this

106 +to be a _bundle list_. The pairs specify information about the bundles

107 +that the client can use to make decisions for which bundles to download

108 +and which to ignore.

109 +

110 +A few keys focus on properties of the list itself.

111 +

112 +bundle.version::

113 + (Required) This value provides a version number for the bundle

114 + list. If a future Git change enables a feature that needs the Git

115 + client to react to a new key in the bundle list file, then this version

116 + will increment. The only current version number is 1, and if any other

117 + value is specified then Git will fail to use this file.

118 +

119 +bundle.mode::

120 + (Required) This value has one of two values: `all` and `any`. When `all`

121 + is specified, then the client should expect to need all of the listed

122 + bundle URIs that match their repository's requirements. When `any` is

123 + specified, then the client should expect that any one of the bundle URIs

124 + that match their repository's requirements will suffice. Typically, the

125 + `any` option is used to list a number of different bundle servers

126 + located in different geographies.

127 +

128 +bundle.heuristic::

129 + If this string-valued key exists, then the bundle list is designed to

130 + work well with incremental `git fetch` commands. The heuristic signals

131 + that there are additional keys available for each bundle that help

132 + determine which subset of bundles the client should download. The only

133 + heuristic currently planned is `creationToken`.

134 +

135 +The remaining keys include an `<id>` segment which is a server-designated

136 +name for each available bundle. The `<id>` must contain only alphanumeric

137 +and `-` characters.

138 +

139 +bundle.<id>.uri::

140 + (Required) This string value is the URI for downloading bundle `<id>`.

141 + If the URI begins with a protocol (`http://` or `https://`) then the URI

142 + is absolute. Otherwise, the URI is interpreted as relative to the URI

143 + used for the bundle list. If the URI begins with `/`, then that relative

144 + path is relative to the domain name used for the bundle list. (This use

145 + of relative paths is intended to make it easier to distribute a set of

146 + bundles across a large number of servers or CDNs with different domain

147 + names.)

148 +

149 +bundle.<id>.filter::

150 + This string value represents an object filter that should also appear in

151 + the header of this bundle. The server uses this value to differentiate

152 + different kinds of bundles from which the client can choose those that

153 + match their object filters.

154 +

155 +bundle.<id>.creationToken::

156 + This value is a nonnegative 64-bit integer used for sorting the bundles

157 + the list. This is used to download a subset of bundles during a fetch

158 + when `bundle.heuristic=creationToken`.

159 +

160 +bundle.<id>.location::

161 + This string value advertises a real-world location from where the bundle

162 + URI is served. This can be used to present the user with an option for

163 + which bundle URI to use or simply as an informative indicator of which

164 + bundle URI was selected by Git. This is only valuable when

165 + `bundle.mode` is `any`.

166 +

167 +Here is an example bundle list using the Git config format:

168 +

169 + [bundle]

170 + version = 1

171 + mode = all

172 + heuristic = creationToken

173 +

174 + [bundle "2022-02-09-1644442601-daily"]

175 + uri = https://bundles.example.com/git/git/2022-02-09-1644442601-daily.bundle

176 + creationToken = 1644442601

177 +

178 + [bundle "2022-02-02-1643842562"]

179 + uri = https://bundles.example.com/git/git/2022-02-02-1643842562.bundle

180 + creationToken = 1643842562

181 +

182 + [bundle "2022-02-09-1644442631-daily-blobless"]

183 + uri = 2022-02-09-1644442631-daily-blobless.bundle

184 + creationToken = 1644442631

185 + filter = blob:none

186 +

187 + [bundle "2022-02-02-1643842568-blobless"]

188 + uri = /git/git/2022-02-02-1643842568-blobless.bundle

189 + creationToken = 1643842568

190 + filter = blob:none

191 +

192 +This example uses `bundle.mode=all` as well as the

193 +`bundle.<id>.creationToken` heuristic. It also uses the `bundle.<id>.filter`

194 +options to present two parallel sets of bundles: one for full clones and

195 +another for blobless partial clones.

196 +

197 +Suppose that this bundle list was found at the URI

198 +`https://bundles.example.com/git/git/` and so the two blobless bundles have

199 +the following fully-expanded URIs:

200 +

201 +* `https://bundles.example.com/git/git/2022-02-09-1644442631-daily-blobless.bundle`

202 +* `https://bundles.example.com/git/git/2022-02-02-1643842568-blobless.bundle`

203 +

204 +Advertising Bundle URIs

205 +-----------------------

206 +

207 +If a user knows a bundle URI for the repository they are cloning, then

208 +they can specify that URI manually through a command-line option. However,

209 +a Git host may want to advertise bundle URIs during the clone operation,

210 +helping users unaware of the feature.

211 +

212 +The only thing required for this feature is that the server can advertise

213 +one or more bundle URIs. This advertisement takes the form of a new

214 +protocol v2 capability specifically for discovering bundle URIs.

215 +

216 +The client could choose an arbitrary bundle URI as an option _or_ select

217 +the URI with best performance by some exploratory checks. It is up to the

218 +bundle provider to decide if having multiple URIs is preferable to a

219 +single URI that is geodistributed through server-side infrastructure.

220 +

221 +Cloning with Bundle URIs

222 +------------------------

223 +

224 +The primary need for bundle URIs is to speed up clones. The Git client

225 +will interact with bundle URIs according to the following flow:

226 +

227 +1. The user specifies a bundle URI with the `--bundle-uri` command-line

228 + option _or_ the client discovers a bundle list advertised by the

229 + Git server.

230 +

231 +2. If the downloaded data from a bundle URI is a bundle, then the client

232 + inspects the bundle headers to check that the prerequisite commit OIDs

233 + are present in the client repository. If some are missing, then the

234 + client delays unbundling until other bundles have been unbundled,

235 + making those OIDs present. When all required OIDs are present, the

236 + client unbundles that data using a refspec. The default refspec is

237 + `+refs/heads/*:refs/bundles/*`, but this can be configured. These refs

238 + are stored so that later `git fetch` negotiations can communicate the

239 + bundled refs as `have`s, reducing the size of the fetch over the Git

240 + protocol. To allow pruning refs from this ref namespace, Git may

241 + introduce a numbered namespace (such as `refs/bundles/<i>/*`) such that

242 + stale bundle refs can be deleted.

243 +

244 +3. If the file is instead a bundle list, then the client inspects the

245 + `bundle.mode` to see if the list is of the `all` or `any` form.

246 +

247 + a. If `bundle.mode=all`, then the client considers all bundle

248 + URIs. The list is reduced based on the `bundle.<id>.filter` options

249 + matching the client repository's partial clone filter. Then, all

250 + bundle URIs are requested. If the `bundle.<id>.creationToken`

251 + heuristic is provided, then the bundles are downloaded in decreasing

252 + order by the creation token, stopping when a bundle has all required

253 + OIDs. The bundles can then be unbundled in increasing creation token

254 + order. The client stores the latest creation token as a heuristic

255 + for avoiding future downloads if the bundle list does not advertise

256 + bundles with larger creation tokens.

257 +

258 + b. If `bundle.mode=any`, then the client can choose any one of the

259 + bundle URIs to inspect. The client can use a variety of ways to

260 + choose among these URIs. The client can also fallback to another URI

261 + if the initial choice fails to return a result.

262 +

263 +Note that during a clone we expect that all bundles will be required, and

264 +heuristics such as `bundle.<uri>.creationToken` can be used to download

265 +bundles in chronological order or in parallel.

266 +

267 +If a given bundle URI is a bundle list with a `bundle.heuristic`

268 +value, then the client can choose to store that URI as its chosen bundle

269 +URI. The client can then navigate directly to that URI during later `git

270 +fetch` calls.

271 +

272 +When downloading bundle URIs, the client can choose to inspect the initial

273 +content before committing to downloading the entire content. This may

274 +provide enough information to determine if the URI is a bundle list or

275 +a bundle. In the case of a bundle, the client may inspect the bundle

276 +header to determine that all advertised tips are already in the client

277 +repository and cancel the remaining download.

278 +

279 +Fetching with Bundle URIs

280 +-------------------------

281 +

282 +When the client fetches new data, it can decide to fetch from bundle

283 +servers before fetching from the origin remote. This could be done via a

284 +command-line option, but it is more likely useful to use a config value

285 +such as the one specified during the clone.

286 +

287 +The fetch operation follows the same procedure to download bundles from a

288 +bundle list (although we do _not_ want to use parallel downloads here). We

289 +expect that the process will end when all prerequisite commit OIDs in a

290 +thin bundle are already in the object database.

291 +

292 +When using the `creationToken` heuristic, the client can avoid downloading

293 +any bundles if their creation tokenss are not larger than the stored

294 +creation token. After fetching new bundles, Git updates this local

295 +creation token.

296 +

297 +If the bundle provider does not provide a heuristic, then the client

298 +should attempt to inspect the bundle headers before downloading the full

299 +bundle data in case the bundle tips already exist in the client

300 +repository.

301 +

302 +Error Conditions

303 +----------------

304 +

305 +If the Git client discovers something unexpected while downloading

306 +information according to a bundle URI or the bundle list found at that

307 +location, then Git can ignore that data and continue as if it was not

308 +given a bundle URI. The remote Git server is the ultimate source of truth,

309 +not the bundle URI.

310 +

311 +Here are a few example error conditions:

312 +

313 +* The client fails to connect with a server at the given URI or a connection

314 + is lost without any chance to recover.

315 +

316 +* The client receives a 400-level response (such as `404 Not Found` or

317 + `401 Not Authorized`). The client should use the credential helper to

318 + find and provide a credential for the URI, but match the semantics of

319 + Git's other HTTP protocols in terms of handling specific 400-level

320 + errors.

321 +

322 +* The server reports any other failure reponse.

323 +

324 +* The client receives data that is not parsable as a bundle or bundle list.

325 +

326 +* A bundle includes a filter that does not match expectations.

327 +

328 +* The client cannot unbundle the bundles because the prerequisite commit OIDs

329 + are not in the object database and there are no more bundles to download.

330 +

331 +There are also situations that could be seen as wasteful, but are not

332 +error conditions:

333 +

334 +* The downloaded bundles contain more information than is requested by

335 + the clone or fetch request. A primary example is if the user requests

336 + a clone with `--single-branch` but downloads bundles that store every

337 + reachable commit from all `refs/heads/*` references. This might be

338 + initially wasteful, but perhaps these objects will become reachable by

339 + a later ref update that the client cares about.

340 +

341 +* A bundle download during a `git fetch` contains objects already in the

342 + object database. This is probably unavoidable if we are using bundles

343 + for fetches, since the client will almost always be slightly ahead of

344 + the bundle servers after performing its "catch-up" fetch to the remote

345 + server. This extra work is most wasteful when the client is fetching

346 + much more frequently than the server is computing bundles, such as if

347 + the client is using hourly prefetches with background maintenance, but

348 + the server is computing bundles weekly. For this reason, the client

349 + should not use bundle URIs for fetch unless the server has explicitly

350 + recommended it through a `bundle.heuristic` value.

351 +

352 +Implementation Plan

353 +-------------------

354 +

355 +This design document is being submitted on its own as an aspirational

356 +document, with the goal of implementing all of the mentioned client

357 +features over the course of several patch series. Here is a potential

358 +outline for submitting these features:

359 +

360 +1. Integrate bundle URIs into `git clone` with a `--bundle-uri` option.

361 + This will include a new `git fetch --bundle-uri` mode for use as the

362 + implementation underneath `git clone`. The initial version here will

363 + expect a single bundle at the given URI.

364 +

365 +2. Implement the ability to parse a bundle list from a bundle URI and

366 + update the `git fetch --bundle-uri` logic to properly distinguish

367 + between `bundle.mode` options. Specifically design the feature so

368 + that the config format parsing feeds a list of key-value pairs into the

369 + bundle list logic.

370 +

371 +3. Create the `bundle-uri` protocol v2 command so Git servers can advertise

372 + bundle URIs using the key-value pairs. Plug into the existing key-value

373 + input to the bundle list logic. Allow `git clone` to discover these

374 + bundle URIs and bootstrap the client repository from the bundle data.

375 + (This choice is an opt-in via a config option and a command-line

376 + option.)

377 +

378 +4. Allow the client to understand the `bundle.flag=forFetch` configuration

379 + and the `bundle.<id>.creationToken` heuristic. When `git clone`

380 + discovers a bundle URI with `bundle.flag=forFetch`, it configures the

381 + client repository to check that bundle URI during later `git fetch <remote>`

382 + commands.

383 +

384 +5. Allow clients to discover bundle URIs during `git fetch` and configure

385 + a bundle URI for later fetches if `bundle.flag=forFetch`.

386 +

387 +6. Implement the "inspect headers" heuristic to reduce data downloads when

388 + the `bundle.<id>.creationToken` heuristic is not available.

389 +

390 +As these features are reviewed, this plan might be updated. We also expect

391 +that new designs will be discovered and implemented as this feature

392 +matures and becomes used in real-world scenarios.

393 +

394 +Related Work: Packfile URIs

395 +---------------------------

396 +

397 +The Git protocol already has a capability where the Git server can list

398 +a set of URLs along with the packfile response when serving a client

399 +request. The client is then expected to download the packfiles at those

400 +locations in order to have a complete understanding of the response.

401 +

402 +This mechanism is used by the Gerrit server (implemented with JGit) and

403 +has been effective at reducing CPU load and improving user performance for

404 +clones.

405 +

406 +A major downside to this mechanism is that the origin server needs to know

407 +_exactly_ what is in those packfiles, and the packfiles need to be available

408 +to the user for some time after the server has responded. This coupling

409 +between the origin and the packfile data is difficult to manage.

410 +

411 +Further, this implementation is extremely hard to make work with fetches.

412 +

413 +Related Work: GVFS Cache Servers

414 +--------------------------------

415 +

416 +The GVFS Protocol [2] is a set of HTTP endpoints designed independently of

417 +the Git project before Git's partial clone was created. One feature of this

418 +protocol is the idea of a "cache server" which can be colocated with build

419 +machines or developer offices to transfer Git data without overloading the

420 +central server.

421 +

422 +The endpoint that VFS for Git is famous for is the `GET /gvfs/objects/{oid}`

423 +endpoint, which allows downloading an object on-demand. This is a critical

424 +piece of the filesystem virtualization of that product.

425 +

426 +However, a more subtle need is the `GET /gvfs/prefetch?lastPackTimestamp=<t>`

427 +endpoint. Given an optional timestamp, the cache server responds with a list

428 +of precomputed packfiles containing the commits and trees that were introduced

429 +in those time intervals.

430 +

431 +The cache server computes these "prefetch" packfiles using the following

432 +strategy:

433 +

434 +1. Every hour, an "hourly" pack is generated with a given timestamp.

435 +2. Nightly, the previous 24 hourly packs are rolled up into a "daily" pack.

436 +3. Nightly, all prefetch packs more than 30 days old are rolled up into

437 + one pack.

438 +

439 +When a user runs `gvfs clone` or `scalar clone` against a repo with cache

440 +servers, the client requests all prefetch packfiles, which is at most

441 +`24 + 30 + 1` packfiles downloading only commits and trees. The client

442 +then follows with a request to the origin server for the references, and

443 +attempts to checkout that tip reference. (There is an extra endpoint that

444 +helps get all reachable trees from a given commit, in case that commit

445 +was not already in a prefetch packfile.)

446 +

447 +During a `git fetch`, a hook requests the prefetch endpoint using the

448 +most-recent timestamp from a previously-downloaded prefetch packfile.

449 +Only the list of packfiles with later timestamps are downloaded. Most

450 +users fetch hourly, so they get at most one hourly prefetch pack. Users

451 +whose machines have been off or otherwise have not fetched in over 30 days

452 +might redownload all prefetch packfiles. This is rare.

453 +

454 +It is important to note that the clients always contact the origin server

455 +for the refs advertisement, so the refs are frequently "ahead" of the

456 +prefetched pack data. The missing objects are downloaded on-demand using

457 +the `GET gvfs/objects/{oid}` requests, when needed by a command such as

458 +`git checkout` or `git log`. Some Git optimizations disable checks that

459 +would cause these on-demand downloads to be too aggressive.

460 +

461 +See Also

462 +--------

463 +

464 +[1] https://lore.kernel.org/git/RFC-cover-00.13-0000000000-20210805T150534Z-avarab@gmail.com/

465 + An earlier RFC for a bundle URI feature.

466 +

467 +[2] https://github.com/microsoft/VFSForGit/blob/master/Protocol.md

468 + The GVFS Protocol

docs: document bundle URI standard