| 1 | Large Object Promisors |
| 2 | ====================== |
| 3 | |
| 4 | Since Git has been created, users have been complaining about issues |
| 5 | with storing large files in Git. Some solutions have been created to |
| 6 | help, but they haven't helped much with some issues. |
| 7 | |
| 8 | Git currently supports multiple promisor remotes, which could help |
| 9 | with some of these remaining issues, but it's very hard to use them to |
| 10 | help, because a number of important features are missing. |
| 11 | |
| 12 | The goal of the effort described in this document is to add these |
| 13 | important features. |
| 14 | |
| 15 | We will call a "Large Object Promisor", or "LOP" in short, a promisor |
| 16 | remote which is used to store only large blobs and which is separate |
| 17 | from the main remote that should store the other Git objects and the |
| 18 | rest of the repos. |
| 19 | |
| 20 | By extension, we will also call "Large Object Promisor", or LOP, the |
| 21 | effort described in this document to add a set of features to make it |
| 22 | easier to handle large blobs/files in Git by using LOPs. |
| 23 | |
| 24 | This effort aims to especially improve things on the server side, and |
| 25 | especially for large blobs that are already compressed in a binary |
| 26 | format. |
| 27 | |
| 28 | This effort aims to provide an alternative to Git LFS |
| 29 | (https://git-lfs.com/) and similar tools like git-annex |
| 30 | (https://git-annex.branchable.com/) for handling large files, even |
| 31 | though a complete alternative would very likely require other efforts |
| 32 | especially on the client side, where it would likely help to implement |
| 33 | a new object representation for large blobs as discussed in: |
| 34 | |
| 35 | https://lore.kernel.org/git/xmqqbkdometi.fsf@gitster.g/ |
| 36 | |
| 37 | Non goals |
| 38 | --------- |
| 39 | |
| 40 | - We will not discuss those client side improvements here, as they |
| 41 | would require changes in different parts of Git than this effort. |
| 42 | + |
| 43 | So we don't pretend to fully replace Git LFS with only this effort, |
| 44 | but we nevertheless believe that it can significantly improve the |
| 45 | current situation on the server side, and that other separate |
| 46 | efforts could also improve the situation on the client side. |
| 47 | |
| 48 | - In the same way, we are not going to discuss all the possible ways |
| 49 | to implement a LOP or their underlying object storage, or to |
| 50 | optimize how LOP works. |
| 51 | + |
| 52 | Our opinion is that the simplest solution for now is for LOPs to use |
| 53 | object storage through a remote helper (see section II.2 below for |
| 54 | more details) to store their objects. So we consider that this is the |
| 55 | default implementation. If there are improvements on top of this, |
| 56 | that's great, but our opinion is that such improvements are not |
| 57 | necessary for LOPs to already be useful. Such improvements are likely |
| 58 | a different technical topic, and can be taken care of separately |
| 59 | anyway. |
| 60 | + |
| 61 | So in particular we are not going to discuss pluggable ODBs or other |
| 62 | object database backends that could chunk large blobs, dedup the |
| 63 | chunks and store them efficiently. Sure, that would be a nice |
| 64 | improvement to store large blobs on the server side, but we believe |
| 65 | it can just be a separate effort as it's also not technically very |
| 66 | related to this effort. |
| 67 | + |
| 68 | We are also not going to discuss data transfer improvements between |
| 69 | LOPs and clients or servers. Sure, there might be some easy and very |
| 70 | effective optimizations there (as we know that objects on LOPs are |
| 71 | very likely incompressible and not deltifying well), but this can be |
| 72 | dealt with separately in a separate effort. |
| 73 | |
| 74 | In other words, the goal of this document is not to talk about all the |
| 75 | possible ways to optimize how Git could handle large blobs, but to |
| 76 | describe how a LOP based solution can already work well and alleviate |
| 77 | a number of current issues in the context of Git clients and servers |
| 78 | sharing Git objects. |
| 79 | |
| 80 | Even if LOPs are used not very efficiently, they can still be useful |
| 81 | and worth using in some cases, as we will see in more details |
| 82 | later in this document: |
| 83 | |
| 84 | - they can make it simpler for clients to use promisor remotes and |
| 85 | therefore avoid fetching a lot of large blobs they might not need |
| 86 | locally, |
| 87 | |
| 88 | - they can make it significantly cheaper or easier for servers to |
| 89 | host a significant part of the current repository content, and |
| 90 | even more to host content with larger blobs or more large blobs |
| 91 | than currently. |
| 92 | |
| 93 | I Issues with the current situation |
| 94 | ----------------------------------- |
| 95 | |
| 96 | - Some statistics made on GitLab repos have shown that more than 75% |
| 97 | of the disk space is used by blobs that are larger than 1MB and |
| 98 | often in a binary format. |
| 99 | |
| 100 | - So even if users could use Git LFS or similar tools to store a lot |
| 101 | of large blobs out of their repos, it's a fact that in practice they |
| 102 | don't do it as much as they probably should. |
| 103 | |
| 104 | - On the server side ideally, the server should be able to decide for |
| 105 | itself how it stores things. It should not depend on users deciding |
| 106 | to use tools like Git LFS on some blobs or not. |
| 107 | |
| 108 | - It's much more expensive to store large blobs that don't delta |
| 109 | compress well on regular fast seeking drives (like SSDs) than on |
| 110 | object storage (like Amazon S3 or GCP Buckets). Using fast drives |
| 111 | for regular Git repos makes sense though, as serving regular Git |
| 112 | content (blobs containing text or code) needs drives where seeking |
| 113 | is fast, but the content is relatively small. On the other hand, |
| 114 | object storage for Git LFS blobs makes sense as seeking speed is not |
| 115 | as important when dealing with large files, while costs are more |
| 116 | important. So the fact that users don't use Git LFS or similar tools |
| 117 | for a significant number of large blobs has likely some bad |
| 118 | consequences on the cost of repo storage for most Git hosting |
| 119 | platforms. |
| 120 | |
| 121 | - Having large blobs handled in the same way as other blobs and Git |
| 122 | objects in Git repos instead of on object storage also has a cost in |
| 123 | increased memory and CPU usage, and therefore decreased performance, |
| 124 | when creating packfiles. (This is because Git tries to use delta |
| 125 | compression or zlib compression which is unlikely to work well on |
| 126 | already compressed binary content.) So it's not just a storage cost |
| 127 | increase. |
| 128 | |
| 129 | - When a large blob has been committed into a repo, it might not be |
| 130 | possible to remove this blob from the repo without rewriting |
| 131 | history, even if the user then decides to use Git LFS or a similar |
| 132 | tool to handle it. |
| 133 | |
| 134 | - In fact Git LFS and similar tools are not very flexible in letting |
| 135 | users change their minds about the blobs they should handle or not. |
| 136 | |
| 137 | - Even when users are using Git LFS or similar tools, they are often |
| 138 | complaining that these tools require significant effort to set up, |
| 139 | learn and use correctly. |
| 140 | |
| 141 | II Main features of the "Large Object Promisors" solution |
| 142 | --------------------------------------------------------- |
| 143 | |
| 144 | The main features below should give a rough overview of how the |
| 145 | solution may work. Details about needed elements can be found in |
| 146 | following sections. |
| 147 | |
| 148 | Even if each feature below is very useful for the full solution, it is |
| 149 | very likely to be also useful on its own in some cases where the full |
| 150 | solution is not required. However, we'll focus primarily on the big |
| 151 | picture here. |
| 152 | |
| 153 | Also each feature doesn't need to be implemented entirely in Git |
| 154 | itself. Some could be scripts, hooks or helpers that are not part of |
| 155 | the Git repo. It would be helpful if those could be shared and |
| 156 | improved on collaboratively though. So we want to encourage sharing |
| 157 | them. |
| 158 | |
| 159 | 1) Large blobs are stored on LOPs |
| 160 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 161 | |
| 162 | Large blobs should be stored on special promisor remotes that we will |
| 163 | call "Large Object Promisors" or LOPs. These LOPs should be additional |
| 164 | remotes dedicated to contain large blobs especially those in binary |
| 165 | format. They should be used along with main remotes that contain the |
| 166 | other objects. |
| 167 | |
| 168 | Note 1 |
| 169 | ^^^^^^ |
| 170 | |
| 171 | To clarify, a LOP is a normal promisor remote, except that: |
| 172 | |
| 173 | - it should store only large blobs, |
| 174 | |
| 175 | - it should be separate from the main remote, so that the main remote |
| 176 | can focus on serving other objects and the rest of the repos (see |
| 177 | feature 4) below) and can use the LOP as a promisor remote for |
| 178 | itself. |
| 179 | |
| 180 | Note 2 |
| 181 | ^^^^^^ |
| 182 | |
| 183 | Git already makes it possible for a main remote to also be a promisor |
| 184 | remote storing both regular objects and large blobs for a client that |
| 185 | clones from it with a filter on blob size. But here we explicitly want |
| 186 | to avoid that. |
| 187 | |
| 188 | Rationale |
| 189 | ^^^^^^^^^ |
| 190 | |
| 191 | LOPs aim to be good at handling large blobs while main remotes are |
| 192 | already good at handling other objects. |
| 193 | |
| 194 | Implementation |
| 195 | ^^^^^^^^^^^^^^ |
| 196 | |
| 197 | Git already has support for multiple promisor remotes, see |
| 198 | link:partial-clone.html#using-many-promisor-remotes[the partial clone documentation]. |
| 199 | |
| 200 | Also, Git already has support for partial clone using a filter on the |
| 201 | size of the blobs (with `git clone --filter=blob:limit=<size>`). Most |
| 202 | of the other main features below are based on these existing features |
| 203 | and are about making them easy and efficient to use for the purpose of |
| 204 | better handling large blobs. |
| 205 | |
| 206 | 2) LOPs can use object storage |
| 207 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 208 | |
| 209 | LOPs can be implemented using object storage, like an Amazon S3 or GCP |
| 210 | Bucket or MinIO (which is open source under the GNU AGPLv3 license) to |
| 211 | actually store the large blobs, and can be accessed through a Git |
| 212 | remote helper (see linkgit:gitremote-helpers[7]) which makes the |
| 213 | underlying object storage appear like a remote to Git. |
| 214 | |
| 215 | Note |
| 216 | ^^^^ |
| 217 | |
| 218 | A LOP can be a promisor remote accessed using a remote helper by |
| 219 | both some clients and the main remote. |
| 220 | |
| 221 | Rationale |
| 222 | ^^^^^^^^^ |
| 223 | |
| 224 | This looks like the simplest way to create LOPs that can cheaply |
| 225 | handle many large blobs. |
| 226 | |
| 227 | Implementation |
| 228 | ^^^^^^^^^^^^^^ |
| 229 | |
| 230 | Remote helpers are quite easy to write as shell scripts, but it might |
| 231 | be more efficient and maintainable to write them using other languages |
| 232 | like Go. |
| 233 | |
| 234 | Some already exist under open source licenses, for example: |
| 235 | |
| 236 | - https://github.com/awslabs/git-remote-s3 |
| 237 | - https://gitlab.com/eric.p.ju/git-remote-gs |
| 238 | |
| 239 | Other ways to implement LOPs are certainly possible, but the goal of |
| 240 | this document is not to discuss how to best implement a LOP or its |
| 241 | underlying object storage (see the "0) Non goals" section above). |
| 242 | |
| 243 | 3) LOP object storage can be Git LFS storage |
| 244 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 245 | |
| 246 | The underlying object storage that a LOP uses could also serve as |
| 247 | storage for large files handled by Git LFS. |
| 248 | |
| 249 | Rationale |
| 250 | ^^^^^^^^^ |
| 251 | |
| 252 | This would simplify the server side if it wants to both use a LOP and |
| 253 | act as a Git LFS server. |
| 254 | |
| 255 | 4) A main remote can offload to a LOP with a configurable threshold |
| 256 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 257 | |
| 258 | On the server side, a main remote should have a way to offload to a |
| 259 | LOP all its blobs with a size over a configurable threshold. |
| 260 | |
| 261 | Rationale |
| 262 | ^^^^^^^^^ |
| 263 | |
| 264 | This makes it easy to set things up and to clean things up. For |
| 265 | example, an admin could use this to manually convert a repo not using |
| 266 | LOPs to a repo using a LOP. On a repo already using a LOP but where |
| 267 | some users would sometimes push large blobs, a cron job could use this |
| 268 | to regularly make sure the large blobs are moved to the LOP. |
| 269 | |
| 270 | Implementation |
| 271 | ^^^^^^^^^^^^^^ |
| 272 | |
| 273 | Using something based on `git repack --filter=...` to separate the |
| 274 | blobs we want to offload from the other Git objects could be a good |
| 275 | idea. The missing part is to connect to the LOP, check if the blobs we |
| 276 | want to offload are already there and if not send them. |
| 277 | |
| 278 | 5) A main remote should try to remain clean from large blobs |
| 279 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 280 | |
| 281 | A main remote should try to avoid containing a lot of oversize |
| 282 | blobs. For that purpose, it should offload as needed to a LOP and it |
| 283 | should have ways to prevent oversize blobs to be fetched, and also |
| 284 | perhaps pushed, into it. |
| 285 | |
| 286 | Rationale |
| 287 | ^^^^^^^^^ |
| 288 | |
| 289 | A main remote containing many oversize blobs would defeat the purpose |
| 290 | of LOPs. |
| 291 | |
| 292 | Implementation |
| 293 | ^^^^^^^^^^^^^^ |
| 294 | |
| 295 | The way to offload to a LOP discussed in 4) above can be used to |
| 296 | regularly offload oversize blobs. About preventing oversize blobs from |
| 297 | being fetched into the repo see 6) below. About preventing oversize |
| 298 | blob pushes, a pre-receive hook could be used. |
| 299 | |
| 300 | Also there are different scenarios in which large blobs could get |
| 301 | fetched into the main remote, for example: |
| 302 | |
| 303 | - A client that doesn't implement the "promisor-remote" protocol |
| 304 | (described in 6) below) clones from the main remote. |
| 305 | |
| 306 | - The main remote gets a request for information about a large blob |
| 307 | and is not able to get that information without fetching the blob |
| 308 | from the LOP. |
| 309 | |
| 310 | It might not be possible to completely prevent all these scenarios |
| 311 | from happening. So the goal here should be to implement features that |
| 312 | make the fetching of large blobs less likely. For example adding a |
| 313 | `remote-object-info` command in the `git cat-file --batch` protocol |
| 314 | and its variants might make it possible for a main repo to respond to |
| 315 | some requests about large blobs without fetching them. |
| 316 | |
| 317 | 6) A protocol negotiation should happen when a client clones |
| 318 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 319 | |
| 320 | When a client clones from a main repo, there should be a protocol |
| 321 | negotiation so that the server can advertise one or more LOPs and so |
| 322 | that the client and the server can discuss if the client could |
| 323 | directly use a LOP the server is advertising. If the client and the |
| 324 | server can agree on that, then the client would be able to get the |
| 325 | large blobs directly from the LOP and the server would not need to |
| 326 | fetch those blobs from the LOP to be able to serve the client. |
| 327 | |
| 328 | Note |
| 329 | ^^^^ |
| 330 | |
| 331 | For fetches instead of clones, a protocol negotiation might not always |
| 332 | happen, see the "What about fetches?" FAQ entry below for details. |
| 333 | |
| 334 | Rationale |
| 335 | ^^^^^^^^^ |
| 336 | |
| 337 | Security, configurability and efficiency of setting things up. |
| 338 | |
| 339 | Implementation |
| 340 | ^^^^^^^^^^^^^^ |
| 341 | |
| 342 | A "promisor-remote" protocol v2 capability looks like a good way to |
| 343 | implement this. The way the client and server use this capability |
| 344 | could be controlled by configuration variables. |
| 345 | |
| 346 | Information that the server could send to the client through that |
| 347 | protocol could be things like: LOP name, LOP URL, filter-spec (for |
| 348 | example `blob:limit=<size>`) or just size limit that should be used as |
| 349 | a filter when cloning, token to be used with the LOP, etc. |
| 350 | |
| 351 | 7) A client can offload to a LOP |
| 352 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 353 | |
| 354 | When a client is using a LOP that is also a LOP of its main remote, |
| 355 | the client should be able to offload some large blobs it has fetched, |
| 356 | but might not need anymore, to the LOP. |
| 357 | |
| 358 | Note |
| 359 | ^^^^ |
| 360 | |
| 361 | It might depend on the context if it should be OK or not for clients |
| 362 | to offload large blobs they have created, instead of fetched, directly |
| 363 | to the LOP without the main remote checking them in some ways |
| 364 | (possibly using hooks or other tools). |
| 365 | |
| 366 | This should be discussed and refined when we get closer to |
| 367 | implementing this feature. |
| 368 | |
| 369 | Rationale |
| 370 | ^^^^^^^^^ |
| 371 | |
| 372 | On the client, the easiest way to deal with unneeded large blobs is to |
| 373 | offload them. |
| 374 | |
| 375 | Implementation |
| 376 | ^^^^^^^^^^^^^^ |
| 377 | |
| 378 | This is very similar to what 4) above is about, except on the client |
| 379 | side instead of the server side. So a good solution to 4) could likely |
| 380 | be adapted to work on the client side too. |
| 381 | |
| 382 | There might be some security issues here, as there is no negotiation, |
| 383 | but they might be mitigated if the client can reuse a token it got |
| 384 | when cloning (see 6) above). Also if the large blobs were fetched from |
| 385 | a LOP, it is likely, and can easily be confirmed, that the LOP still |
| 386 | has them, so that they can just be removed from the client. |
| 387 | |
| 388 | III Benefits of using LOPs |
| 389 | -------------------------- |
| 390 | |
| 391 | Many benefits are related to the issues discussed in "I) Issues with |
| 392 | the current situation" above: |
| 393 | |
| 394 | - No need to rewrite history when deciding which blobs are worth |
| 395 | handling separately than other objects, or when moving or removing |
| 396 | the threshold. |
| 397 | |
| 398 | - If the protocol between client and server is developed and secured |
| 399 | enough, then many details might be setup on the server side only and |
| 400 | all the clients could then easily get all the configuration |
| 401 | information and use it to set themselves up mostly automatically. |
| 402 | |
| 403 | - Storage costs benefits on the server side. |
| 404 | |
| 405 | - Reduced memory and CPU needs on main remotes on the server side. |
| 406 | |
| 407 | - Reduced storage needs on the client side. |
| 408 | |
| 409 | IV FAQ |
| 410 | ------ |
| 411 | |
| 412 | What about using multiple LOPs on the server and client side? |
| 413 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 414 | |
| 415 | That could perhaps be useful in some cases, but for now it's more |
| 416 | likely that in most cases a single LOP will be advertised by the |
| 417 | server and should be used by the client. |
| 418 | |
| 419 | A case where it could be useful for a server to advertise multiple |
| 420 | LOPs is if a LOP is better for some users while a different LOP is |
| 421 | better for other users. For example some clients might have a better |
| 422 | connection to a LOP than others. |
| 423 | |
| 424 | In those cases it's the responsibility of the server to have some |
| 425 | documentation to help clients. It could say for example something like |
| 426 | "Users in this part of the world might want to pick only LOP A as it |
| 427 | is likely to be better connected to them, while users in other parts |
| 428 | of the world should pick only LOP B for the same reason." |
| 429 | |
| 430 | When should we trust or not trust the LOPs advertised by the server? |
| 431 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 432 | |
| 433 | In some contexts, like in corporate setup where the server and all the |
| 434 | clients are parts of an internal network in a company where admins |
| 435 | have all the rights on every system, it's OK, and perhaps even a good |
| 436 | thing, if the clients fully trust the server, as it can help ensure |
| 437 | that all the clients are on the same page. |
| 438 | |
| 439 | There are also contexts in which clients trust a code hosting platform |
| 440 | serving them some repos, but might not fully trust other users |
| 441 | managing or contributing to some of these repos. For example, the code |
| 442 | hosting platform could have hooks in place to check that any object it |
| 443 | receives doesn't contain malware or otherwise bad content. In this |
| 444 | case it might be OK for the client to use a main remote and its LOP if |
| 445 | they are both hosted by the code hosting platform, but not if the LOP |
| 446 | is hosted elsewhere (where the content is not checked). |
| 447 | |
| 448 | In other contexts, a client should just not trust a server. |
| 449 | |
| 450 | So there should be different ways to configure how the client should |
| 451 | behave when a server advertises a LOP to it at clone time. |
| 452 | |
| 453 | As the basic elements that a server can advertise about a LOP are a |
| 454 | LOP name and a LOP URL, the client should base its decision about |
| 455 | accepting a LOP on these elements. |
| 456 | |
| 457 | One simple way to be very strict in the LOP it accepts is for example |
| 458 | for the client to check that the LOP is already configured on the |
| 459 | client with the same name and URL as what the server advertises. |
| 460 | |
| 461 | In general default and "safe" settings should require that the LOP are |
| 462 | configured on the client separately from the "promisor-remote" |
| 463 | protocol and that the client accepts a LOP only when information about |
| 464 | it from the protocol matches what has been already configured |
| 465 | separately. |
| 466 | |
| 467 | What about LOP names? |
| 468 | ~~~~~~~~~~~~~~~~~~~~~ |
| 469 | |
| 470 | In some contexts, for example if the clients sometimes fetch from each |
| 471 | other, it can be a good idea for all the clients to use the same names |
| 472 | for all the remotes they use, including LOPs. |
| 473 | |
| 474 | In other contexts, each client might want to be able to give the name |
| 475 | it wants to each remote, including each LOP, it interacts with. |
| 476 | |
| 477 | So there should be different ways to configure how the client accepts |
| 478 | or not the LOP name the server advertises. |
| 479 | |
| 480 | If a default or "safe" setting is used, then as such a setting should |
| 481 | require that the LOP be configured separately, then the name would be |
| 482 | configured separately and there is no risk that the server could |
| 483 | dictate a name to a client. |
| 484 | |
| 485 | Could the main remote be bogged down by old or paranoid clients? |
| 486 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 487 | |
| 488 | Yes, it could happen if there are too many clients that are either |
| 489 | unwilling to trust the main remote or that just don't implement the |
| 490 | "promisor-remote" protocol because they are too old or not fully |
| 491 | compatible with the 'git' client. |
| 492 | |
| 493 | When serving such a client, the main remote has no other choice than |
| 494 | to first fetch from its LOP, to then be able to provide to the client |
| 495 | everything it requested. So the main remote, even if it has cleanup |
| 496 | mechanisms (see section II.4 above), would be burdened at least |
| 497 | temporarily with the large blobs it had to fetch from its LOP. |
| 498 | |
| 499 | Not behaving like this would be breaking backward compatibility, and |
| 500 | could be seen as segregating clients. For example, it might be |
| 501 | possible to implement a special mode that allows the server to just |
| 502 | reject clients that don't implement the "promisor-remote" protocol or |
| 503 | aren't willing to trust the main remote. This mode might be useful in |
| 504 | a special context like a corporate environment. There is no plan to |
| 505 | implement such a mode though, and this should be discussed separately |
| 506 | later anyway. |
| 507 | |
| 508 | A better way to proceed is probably for the main remote to show a |
| 509 | message telling clients that don't implement the protocol or are |
| 510 | unwilling to accept the advertised LOP(s) that they would get faster |
| 511 | clone and fetches by upgrading client software or properly setting |
| 512 | them up to accept LOP(s). |
| 513 | |
| 514 | Waiting for clients to upgrade, monitoring these upgrades and limiting |
| 515 | the use of LOPs to repos that are not very frequently accessed might |
| 516 | be other good ways to make sure that some benefits are still reaped |
| 517 | from LOPs. Over time, as more and more clients upgrade and benefit |
| 518 | from LOPs, using them in more and more frequently accessed repos will |
| 519 | become worth it. |
| 520 | |
| 521 | Corporate environments, where it might be easier to make sure that all |
| 522 | the clients are up-to-date and properly configured, could hopefully |
| 523 | benefit more and earlier from using LOPs. |
| 524 | |
| 525 | What about fetches? |
| 526 | ~~~~~~~~~~~~~~~~~~~ |
| 527 | |
| 528 | There are different kinds of fetches. A regular fetch happens when |
| 529 | some refs have been updated on the server and the client wants the ref |
| 530 | updates and possibly the new objects added with them. A "backfill" or |
| 531 | "lazy" fetch, on the contrary, happens when the client needs to use |
| 532 | some objects it already knows about but doesn't have because they are |
| 533 | on a promisor remote. |
| 534 | |
| 535 | Regular fetch |
| 536 | ^^^^^^^^^^^^^ |
| 537 | |
| 538 | In a regular fetch, the client will contact the main remote and a |
| 539 | protocol negotiation will happen between them. It's a good thing that |
| 540 | a protocol negotiation happens every time, as the configuration on the |
| 541 | client or the main remote could have changed since the previous |
| 542 | protocol negotiation. In this case, the new protocol negotiation |
| 543 | should ensure that the new fetch will happen in a way that satisfies |
| 544 | the new configuration of both the client and the server. |
| 545 | |
| 546 | In most cases though, the configurations on the client and the main |
| 547 | remote will not have changed between 2 fetches or between the initial |
| 548 | clone and a subsequent fetch. This means that the result of a new |
| 549 | protocol negotiation will be the same as the previous result, so the |
| 550 | new fetch will happen in the same way as the previous clone or fetch, |
| 551 | using, or not using, the same LOP(s) as last time. |
| 552 | |
| 553 | "Backfill" or "lazy" fetch |
| 554 | ^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 555 | |
| 556 | When there is a backfill fetch, the client doesn't necessarily contact |
| 557 | the main remote first. It will try to fetch from its promisor remotes |
| 558 | in the order they appear in the config file, except that a remote |
| 559 | configured using the `extensions.partialClone` config variable will be |
| 560 | tried last. See |
| 561 | link:partial-clone.html#using-many-promisor-remotes[the partial clone documentation]. |
| 562 | |
| 563 | This is not new with this effort. In fact this is how multiple remotes |
| 564 | have already been working for around 5 years. |
| 565 | |
| 566 | When using LOPs, having the main remote configured using |
| 567 | `extensions.partialClone`, so it's tried last, makes sense, as missing |
| 568 | objects should only be large blobs that are on LOPs. |
| 569 | |
| 570 | This means that a protocol negotiation will likely not happen as the |
| 571 | missing objects will be fetched from the LOPs, and then there will be |
| 572 | nothing left to fetch from the main remote. |
| 573 | |
| 574 | To secure that, it could be a good idea for LOPs to require a token |
| 575 | from the client when it fetches from them. The client could get the |
| 576 | token when performing a protocol negotiation with the main remote (see |
| 577 | section II.6 above). |
| 578 | |
| 579 | V Future improvements |
| 580 | --------------------- |
| 581 | |
| 582 | It is expected that at the beginning using LOPs will be mostly worth |
| 583 | it either in a corporate context where the Git version that clients |
| 584 | use can easily be controlled, or on repos that are infrequently |
| 585 | accessed. (See the "Could the main remote be bogged down by old or |
| 586 | paranoid clients?" section in the FAQ above.) |
| 587 | |
| 588 | Over time, as more and more clients upgrade to a version that |
| 589 | implements the "promisor-remote" protocol v2 capability described |
| 590 | above in section II.6), it will be worth it to use LOPs more widely. |
| 591 | |
| 592 | A lot of improvements may also help using LOPs more widely. Some of |
| 593 | these improvements are part of the scope of this document like the |
| 594 | following: |
| 595 | |
| 596 | - Implementing a "remote-object-info" command in the |
| 597 | `git cat-file --batch` protocol and its variants to allow main |
| 598 | remotes to respond to requests about large blobs without fetching |
| 599 | them. (Eric Ju has started working on this based on previous work |
| 600 | by Calvin Wan.) |
| 601 | |
| 602 | - Creating better cleanup and offload mechanisms for main remotes |
| 603 | and clients to prevent accumulation of large blobs. |
| 604 | |
| 605 | - Developing more sophisticated protocol negotiation capabilities |
| 606 | between clients and servers for handling LOPs, for example adding |
| 607 | a filter-spec (e.g., blob:limit=<size>) or size limit for |
| 608 | filtering when cloning, or adding a token for LOP authentication. |
| 609 | |
| 610 | - Improving security measures for LOP access, particularly around |
| 611 | token handling and authentication. |
| 612 | |
| 613 | - Developing standardized ways to configure and manage multiple LOPs |
| 614 | across different environments. Especially in the case where |
| 615 | different LOPs serve the same content to clients in different |
| 616 | geographical locations, there is a need for replication or |
| 617 | synchronization between LOPs. |
| 618 | |
| 619 | Some improvements, including some that have been mentioned in the "0) |
| 620 | Non Goals" section of this document, are out of the scope of this |
| 621 | document: |
| 622 | |
| 623 | - Implementing a new object representation for large blobs on the |
| 624 | client side. |
| 625 | |
| 626 | - Developing pluggable ODBs or other object database backends that |
| 627 | could chunk large blobs, dedup the chunks and store them |
| 628 | efficiently. |
| 629 | |
| 630 | - Optimizing data transfer between LOPs and clients/servers, |
| 631 | particularly for incompressible and non-deltifying content. |
| 632 | |
| 633 | - Creating improved client side tools for managing large objects |
| 634 | more effectively, for example tools for migrating from Git LFS or |
| 635 | git-annex, or tools to find which objects could be offloaded and |
| 636 | how much disk space could be reclaimed by offloading them. |
| 637 | |
| 638 | Some improvements could be seen as part of the scope of this document, |
| 639 | but might already have their own separate projects from the Git |
| 640 | project, like: |
| 641 | |
| 642 | - Improving existing remote helpers to access object storage or |
| 643 | developing new ones. |
| 644 | |
| 645 | - Improving existing object storage solutions or developing new |
| 646 | ones. |
| 647 | |
| 648 | Even though all the above improvements may help, this document and the |
| 649 | LOP effort should try to focus, at least first, on a relatively small |
| 650 | number of improvements mostly those that are in its current scope. |
| 651 | |
| 652 | For example introducing pluggable ODBs and a new object database |
| 653 | backend is likely a multi-year effort on its own that can happen |
| 654 | separately in parallel. It has different technical requirements, |
| 655 | touches other part of the Git code base and should have its own design |
| 656 | document(s). |