@matheusbsbs / matheus-bagliano / commits / 5040f9f164

doc: add technical design doc for large object promisors

Let's add a design doc about how we could improve handling liarge blobs using "Large Object Promisors" (LOPs). It's a set of features with the goal of using special dedicated promisor remotes to store large blobs, and having them accessed directly by main remotes and clients. Signed-off-by: Christian Couder <chriscool@tuxfamily.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Christian Couder committed Feb 18, 2025 at 12:32 UTC 5040f9f164ea7d633237e52bed311d1e9773c78e
1 file changed +656
Documentation/technical/large-object-promisors.txt new
+656
@@ -0,0 +1,656 @@
1 +Large Object Promisors
2 +======================
3 +
4 +Since Git has been created, users have been complaining about issues
5 +with storing large files in Git. Some solutions have been created to
6 +help, but they haven't helped much with some issues.
7 +
8 +Git currently supports multiple promisor remotes, which could help
9 +with some of these remaining issues, but it's very hard to use them to
10 +help, because a number of important features are missing.
11 +
12 +The goal of the effort described in this document is to add these
13 +important features.
14 +
15 +We will call a "Large Object Promisor", or "LOP" in short, a promisor
16 +remote which is used to store only large blobs and which is separate
17 +from the main remote that should store the other Git objects and the
18 +rest of the repos.
19 +
20 +By extension, we will also call "Large Object Promisor", or LOP, the
21 +effort described in this document to add a set of features to make it
22 +easier to handle large blobs/files in Git by using LOPs.
23 +
24 +This effort aims to especially improve things on the server side, and
25 +especially for large blobs that are already compressed in a binary
26 +format.
27 +
28 +This effort aims to provide an alternative to Git LFS
29 +(https://git-lfs.com/) and similar tools like git-annex
30 +(https://git-annex.branchable.com/) for handling large files, even
31 +though a complete alternative would very likely require other efforts
32 +especially on the client side, where it would likely help to implement
33 +a new object representation for large blobs as discussed in:
34 +
35 +https://lore.kernel.org/git/xmqqbkdometi.fsf@gitster.g/
36 +
37 +0) Non goals
38 +------------
39 +
40 +- We will not discuss those client side improvements here, as they
41 + would require changes in different parts of Git than this effort.
42 ++
43 +So we don't pretend to fully replace Git LFS with only this effort,
44 +but we nevertheless believe that it can significantly improve the
45 +current situation on the server side, and that other separate
46 +efforts could also improve the situation on the client side.
47 +
48 +- In the same way, we are not going to discuss all the possible ways
49 + to implement a LOP or their underlying object storage, or to
50 + optimize how LOP works.
51 ++
52 +Our opinion is that the simplest solution for now is for LOPs to use
53 +object storage through a remote helper (see section II.2 below for
54 +more details) to store their objects. So we consider that this is the
55 +default implementation. If there are improvements on top of this,
56 +that's great, but our opinion is that such improvements are not
57 +necessary for LOPs to already be useful. Such improvements are likely
58 +a different technical topic, and can be taken care of separately
59 +anyway.
60 ++
61 +So in particular we are not going to discuss pluggable ODBs or other
62 +object database backends that could chunk large blobs, dedup the
63 +chunks and store them efficiently. Sure, that would be a nice
64 +improvement to store large blobs on the server side, but we believe
65 +it can just be a separate effort as it's also not technically very
66 +related to this effort.
67 ++
68 +We are also not going to discuss data transfer improvements between
69 +LOPs and clients or servers. Sure, there might be some easy and very
70 +effective optimizations there (as we know that objects on LOPs are
71 +very likely incompressible and not deltifying well), but this can be
72 +dealt with separately in a separate effort.
73 +
74 +In other words, the goal of this document is not to talk about all the
75 +possible ways to optimize how Git could handle large blobs, but to
76 +describe how a LOP based solution can already work well and alleviate
77 +a number of current issues in the context of Git clients and servers
78 +sharing Git objects.
79 +
80 +Even if LOPs are used not very efficiently, they can still be useful
81 +and worth using in some cases, as we will see in more details
82 +later in this document:
83 +
84 + - they can make it simpler for clients to use promisor remotes and
85 + therefore avoid fetching a lot of large blobs they might not need
86 + locally,
87 +
88 + - they can make it significantly cheaper or easier for servers to
89 + host a significant part of the current repository content, and
90 + even more to host content with larger blobs or more large blobs
91 + than currently.
92 +
93 +I) Issues with the current situation
94 +------------------------------------
95 +
96 +- Some statistics made on GitLab repos have shown that more than 75%
97 + of the disk space is used by blobs that are larger than 1MB and
98 + often in a binary format.
99 +
100 +- So even if users could use Git LFS or similar tools to store a lot
101 + of large blobs out of their repos, it's a fact that in practice they
102 + don't do it as much as they probably should.
103 +
104 +- On the server side ideally, the server should be able to decide for
105 + itself how it stores things. It should not depend on users deciding
106 + to use tools like Git LFS on some blobs or not.
107 +
108 +- It's much more expensive to store large blobs that don't delta
109 + compress well on regular fast seeking drives (like SSDs) than on
110 + object storage (like Amazon S3 or GCP Buckets). Using fast drives
111 + for regular Git repos makes sense though, as serving regular Git
112 + content (blobs containing text or code) needs drives where seeking
113 + is fast, but the content is relatively small. On the other hand,
114 + object storage for Git LFS blobs makes sense as seeking speed is not
115 + as important when dealing with large files, while costs are more
116 + important. So the fact that users don't use Git LFS or similar tools
117 + for a significant number of large blobs has likely some bad
118 + consequences on the cost of repo storage for most Git hosting
119 + platforms.
120 +
121 +- Having large blobs handled in the same way as other blobs and Git
122 + objects in Git repos instead of on object storage also has a cost in
123 + increased memory and CPU usage, and therefore decreased performance,
124 + when creating packfiles. (This is because Git tries to use delta
125 + compression or zlib compression which is unlikely to work well on
126 + already compressed binary content.) So it's not just a storage cost
127 + increase.
128 +
129 +- When a large blob has been committed into a repo, it might not be
130 + possible to remove this blob from the repo without rewriting
131 + history, even if the user then decides to use Git LFS or a similar
132 + tool to handle it.
133 +
134 +- In fact Git LFS and similar tools are not very flexible in letting
135 + users change their minds about the blobs they should handle or not.
136 +
137 +- Even when users are using Git LFS or similar tools, they are often
138 + complaining that these tools require significant effort to set up,
139 + learn and use correctly.
140 +
141 +II) Main features of the "Large Object Promisors" solution
142 +----------------------------------------------------------
143 +
144 +The main features below should give a rough overview of how the
145 +solution may work. Details about needed elements can be found in
146 +following sections.
147 +
148 +Even if each feature below is very useful for the full solution, it is
149 +very likely to be also useful on its own in some cases where the full
150 +solution is not required. However, we'll focus primarily on the big
151 +picture here.
152 +
153 +Also each feature doesn't need to be implemented entirely in Git
154 +itself. Some could be scripts, hooks or helpers that are not part of
155 +the Git repo. It would be helpful if those could be shared and
156 +improved on collaboratively though. So we want to encourage sharing
157 +them.
158 +
159 +1) Large blobs are stored on LOPs
160 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
161 +
162 +Large blobs should be stored on special promisor remotes that we will
163 +call "Large Object Promisors" or LOPs. These LOPs should be additional
164 +remotes dedicated to contain large blobs especially those in binary
165 +format. They should be used along with main remotes that contain the
166 +other objects.
167 +
168 +Note 1
169 +++++++
170 +
171 +To clarify, a LOP is a normal promisor remote, except that:
172 +
173 +- it should store only large blobs,
174 +
175 +- it should be separate from the main remote, so that the main remote
176 + can focus on serving other objects and the rest of the repos (see
177 + feature 4) below) and can use the LOP as a promisor remote for
178 + itself.
179 +
180 +Note 2
181 +++++++
182 +
183 +Git already makes it possible for a main remote to also be a promisor
184 +remote storing both regular objects and large blobs for a client that
185 +clones from it with a filter on blob size. But here we explicitly want
186 +to avoid that.
187 +
188 +Rationale
189 ++++++++++
190 +
191 +LOPs aim to be good at handling large blobs while main remotes are
192 +already good at handling other objects.
193 +
194 +Implementation
195 +++++++++++++++
196 +
197 +Git already has support for multiple promisor remotes, see
198 +link:partial-clone.html#using-many-promisor-remotes[the partial clone documentation].
199 +
200 +Also, Git already has support for partial clone using a filter on the
201 +size of the blobs (with `git clone --filter=blob:limit=<size>`). Most
202 +of the other main features below are based on these existing features
203 +and are about making them easy and efficient to use for the purpose of
204 +better handling large blobs.
205 +
206 +2) LOPs can use object storage
207 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
208 +
209 +LOPs can be implemented using object storage, like an Amazon S3 or GCP
210 +Bucket or MinIO (which is open source under the GNU AGPLv3 license) to
211 +actually store the large blobs, and can be accessed through a Git
212 +remote helper (see linkgit:gitremote-helpers[7]) which makes the
213 +underlying object storage appear like a remote to Git.
214 +
215 +Note
216 +++++
217 +
218 +A LOP can be a promisor remote accessed using a remote helper by
219 +both some clients and the main remote.
220 +
221 +Rationale
222 ++++++++++
223 +
224 +This looks like the simplest way to create LOPs that can cheaply
225 +handle many large blobs.
226 +
227 +Implementation
228 +++++++++++++++
229 +
230 +Remote helpers are quite easy to write as shell scripts, but it might
231 +be more efficient and maintainable to write them using other languages
232 +like Go.
233 +
234 +Some already exist under open source licenses, for example:
235 +
236 + - https://github.com/awslabs/git-remote-s3
237 + - https://gitlab.com/eric.p.ju/git-remote-gs
238 +
239 +Other ways to implement LOPs are certainly possible, but the goal of
240 +this document is not to discuss how to best implement a LOP or its
241 +underlying object storage (see the "0) Non goals" section above).
242 +
243 +3) LOP object storage can be Git LFS storage
244 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
245 +
246 +The underlying object storage that a LOP uses could also serve as
247 +storage for large files handled by Git LFS.
248 +
249 +Rationale
250 ++++++++++
251 +
252 +This would simplify the server side if it wants to both use a LOP and
253 +act as a Git LFS server.
254 +
255 +4) A main remote can offload to a LOP with a configurable threshold
256 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
257 +
258 +On the server side, a main remote should have a way to offload to a
259 +LOP all its blobs with a size over a configurable threshold.
260 +
261 +Rationale
262 ++++++++++
263 +
264 +This makes it easy to set things up and to clean things up. For
265 +example, an admin could use this to manually convert a repo not using
266 +LOPs to a repo using a LOP. On a repo already using a LOP but where
267 +some users would sometimes push large blobs, a cron job could use this
268 +to regularly make sure the large blobs are moved to the LOP.
269 +
270 +Implementation
271 +++++++++++++++
272 +
273 +Using something based on `git repack --filter=...` to separate the
274 +blobs we want to offload from the other Git objects could be a good
275 +idea. The missing part is to connect to the LOP, check if the blobs we
276 +want to offload are already there and if not send them.
277 +
278 +5) A main remote should try to remain clean from large blobs
279 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
280 +
281 +A main remote should try to avoid containing a lot of oversize
282 +blobs. For that purpose, it should offload as needed to a LOP and it
283 +should have ways to prevent oversize blobs to be fetched, and also
284 +perhaps pushed, into it.
285 +
286 +Rationale
287 ++++++++++
288 +
289 +A main remote containing many oversize blobs would defeat the purpose
290 +of LOPs.
291 +
292 +Implementation
293 +++++++++++++++
294 +
295 +The way to offload to a LOP discussed in 4) above can be used to
296 +regularly offload oversize blobs. About preventing oversize blobs from
297 +being fetched into the repo see 6) below. About preventing oversize
298 +blob pushes, a pre-receive hook could be used.
299 +
300 +Also there are different scenarios in which large blobs could get
301 +fetched into the main remote, for example:
302 +
303 +- A client that doesn't implement the "promisor-remote" protocol
304 + (described in 6) below) clones from the main remote.
305 +
306 +- The main remote gets a request for information about a large blob
307 + and is not able to get that information without fetching the blob
308 + from the LOP.
309 +
310 +It might not be possible to completely prevent all these scenarios
311 +from happening. So the goal here should be to implement features that
312 +make the fetching of large blobs less likely. For example adding a
313 +`remote-object-info` command in the `git cat-file --batch` protocol
314 +and its variants might make it possible for a main repo to respond to
315 +some requests about large blobs without fetching them.
316 +
317 +6) A protocol negotiation should happen when a client clones
318 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
319 +
320 +When a client clones from a main repo, there should be a protocol
321 +negotiation so that the server can advertise one or more LOPs and so
322 +that the client and the server can discuss if the client could
323 +directly use a LOP the server is advertising. If the client and the
324 +server can agree on that, then the client would be able to get the
325 +large blobs directly from the LOP and the server would not need to
326 +fetch those blobs from the LOP to be able to serve the client.
327 +
328 +Note
329 +++++
330 +
331 +For fetches instead of clones, a protocol negotiation might not always
332 +happen, see the "What about fetches?" FAQ entry below for details.
333 +
334 +Rationale
335 ++++++++++
336 +
337 +Security, configurability and efficiency of setting things up.
338 +
339 +Implementation
340 +++++++++++++++
341 +
342 +A "promisor-remote" protocol v2 capability looks like a good way to
343 +implement this. The way the client and server use this capability
344 +could be controlled by configuration variables.
345 +
346 +Information that the server could send to the client through that
347 +protocol could be things like: LOP name, LOP URL, filter-spec (for
348 +example `blob:limit=<size>`) or just size limit that should be used as
349 +a filter when cloning, token to be used with the LOP, etc.
350 +
351 +7) A client can offload to a LOP
352 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
353 +
354 +When a client is using a LOP that is also a LOP of its main remote,
355 +the client should be able to offload some large blobs it has fetched,
356 +but might not need anymore, to the LOP.
357 +
358 +Note
359 +++++
360 +
361 +It might depend on the context if it should be OK or not for clients
362 +to offload large blobs they have created, instead of fetched, directly
363 +to the LOP without the main remote checking them in some ways
364 +(possibly using hooks or other tools).
365 +
366 +This should be discussed and refined when we get closer to
367 +implementing this feature.
368 +
369 +Rationale
370 ++++++++++
371 +
372 +On the client, the easiest way to deal with unneeded large blobs is to
373 +offload them.
374 +
375 +Implementation
376 +++++++++++++++
377 +
378 +This is very similar to what 4) above is about, except on the client
379 +side instead of the server side. So a good solution to 4) could likely
380 +be adapted to work on the client side too.
381 +
382 +There might be some security issues here, as there is no negotiation,
383 +but they might be mitigated if the client can reuse a token it got
384 +when cloning (see 6) above). Also if the large blobs were fetched from
385 +a LOP, it is likely, and can easily be confirmed, that the LOP still
386 +has them, so that they can just be removed from the client.
387 +
388 +III) Benefits of using LOPs
389 +---------------------------
390 +
391 +Many benefits are related to the issues discussed in "I) Issues with
392 +the current situation" above:
393 +
394 +- No need to rewrite history when deciding which blobs are worth
395 + handling separately than other objects, or when moving or removing
396 + the threshold.
397 +
398 +- If the protocol between client and server is developed and secured
399 + enough, then many details might be setup on the server side only and
400 + all the clients could then easily get all the configuration
401 + information and use it to set themselves up mostly automatically.
402 +
403 +- Storage costs benefits on the server side.
404 +
405 +- Reduced memory and CPU needs on main remotes on the server side.
406 +
407 +- Reduced storage needs on the client side.
408 +
409 +IV) FAQ
410 +-------
411 +
412 +What about using multiple LOPs on the server and client side?
413 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
414 +
415 +That could perhaps be useful in some cases, but for now it's more
416 +likely that in most cases a single LOP will be advertised by the
417 +server and should be used by the client.
418 +
419 +A case where it could be useful for a server to advertise multiple
420 +LOPs is if a LOP is better for some users while a different LOP is
421 +better for other users. For example some clients might have a better
422 +connection to a LOP than others.
423 +
424 +In those cases it's the responsibility of the server to have some
425 +documentation to help clients. It could say for example something like
426 +"Users in this part of the world might want to pick only LOP A as it
427 +is likely to be better connected to them, while users in other parts
428 +of the world should pick only LOP B for the same reason."
429 +
430 +When should we trust or not trust the LOPs advertised by the server?
431 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
432 +
433 +In some contexts, like in corporate setup where the server and all the
434 +clients are parts of an internal network in a company where admins
435 +have all the rights on every system, it's OK, and perhaps even a good
436 +thing, if the clients fully trust the server, as it can help ensure
437 +that all the clients are on the same page.
438 +
439 +There are also contexts in which clients trust a code hosting platform
440 +serving them some repos, but might not fully trust other users
441 +managing or contributing to some of these repos. For example, the code
442 +hosting platform could have hooks in place to check that any object it
443 +receives doesn't contain malware or otherwise bad content. In this
444 +case it might be OK for the client to use a main remote and its LOP if
445 +they are both hosted by the code hosting platform, but not if the LOP
446 +is hosted elsewhere (where the content is not checked).
447 +
448 +In other contexts, a client should just not trust a server.
449 +
450 +So there should be different ways to configure how the client should
451 +behave when a server advertises a LOP to it at clone time.
452 +
453 +As the basic elements that a server can advertise about a LOP are a
454 +LOP name and a LOP URL, the client should base its decision about
455 +accepting a LOP on these elements.
456 +
457 +One simple way to be very strict in the LOP it accepts is for example
458 +for the client to check that the LOP is already configured on the
459 +client with the same name and URL as what the server advertises.
460 +
461 +In general default and "safe" settings should require that the LOP are
462 +configured on the client separately from the "promisor-remote"
463 +protocol and that the client accepts a LOP only when information about
464 +it from the protocol matches what has been already configured
465 +separately.
466 +
467 +What about LOP names?
468 +~~~~~~~~~~~~~~~~~~~~~
469 +
470 +In some contexts, for example if the clients sometimes fetch from each
471 +other, it can be a good idea for all the clients to use the same names
472 +for all the remotes they use, including LOPs.
473 +
474 +In other contexts, each client might want to be able to give the name
475 +it wants to each remote, including each LOP, it interacts with.
476 +
477 +So there should be different ways to configure how the client accepts
478 +or not the LOP name the server advertises.
479 +
480 +If a default or "safe" setting is used, then as such a setting should
481 +require that the LOP be configured separately, then the name would be
482 +configured separately and there is no risk that the server could
483 +dictate a name to a client.
484 +
485 +Could the main remote be bogged down by old or paranoid clients?
486 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
487 +
488 +Yes, it could happen if there are too many clients that are either
489 +unwilling to trust the main remote or that just don't implement the
490 +"promisor-remote" protocol because they are too old or not fully
491 +compatible with the 'git' client.
492 +
493 +When serving such a client, the main remote has no other choice than
494 +to first fetch from its LOP, to then be able to provide to the client
495 +everything it requested. So the main remote, even if it has cleanup
496 +mechanisms (see section II.4 above), would be burdened at least
497 +temporarily with the large blobs it had to fetch from its LOP.
498 +
499 +Not behaving like this would be breaking backward compatibility, and
500 +could be seen as segregating clients. For example, it might be
501 +possible to implement a special mode that allows the server to just
502 +reject clients that don't implement the "promisor-remote" protocol or
503 +aren't willing to trust the main remote. This mode might be useful in
504 +a special context like a corporate environment. There is no plan to
505 +implement such a mode though, and this should be discussed separately
506 +later anyway.
507 +
508 +A better way to proceed is probably for the main remote to show a
509 +message telling clients that don't implement the protocol or are
510 +unwilling to accept the advertised LOP(s) that they would get faster
511 +clone and fetches by upgrading client software or properly setting
512 +them up to accept LOP(s).
513 +
514 +Waiting for clients to upgrade, monitoring these upgrades and limiting
515 +the use of LOPs to repos that are not very frequently accessed might
516 +be other good ways to make sure that some benefits are still reaped
517 +from LOPs. Over time, as more and more clients upgrade and benefit
518 +from LOPs, using them in more and more frequently accessed repos will
519 +become worth it.
520 +
521 +Corporate environments, where it might be easier to make sure that all
522 +the clients are up-to-date and properly configured, could hopefully
523 +benefit more and earlier from using LOPs.
524 +
525 +What about fetches?
526 +~~~~~~~~~~~~~~~~~~~
527 +
528 +There are different kinds of fetches. A regular fetch happens when
529 +some refs have been updated on the server and the client wants the ref
530 +updates and possibly the new objects added with them. A "backfill" or
531 +"lazy" fetch, on the contrary, happens when the client needs to use
532 +some objects it already knows about but doesn't have because they are
533 +on a promisor remote.
534 +
535 +Regular fetch
536 ++++++++++++++
537 +
538 +In a regular fetch, the client will contact the main remote and a
539 +protocol negotiation will happen between them. It's a good thing that
540 +a protocol negotiation happens every time, as the configuration on the
541 +client or the main remote could have changed since the previous
542 +protocol negotiation. In this case, the new protocol negotiation
543 +should ensure that the new fetch will happen in a way that satisfies
544 +the new configuration of both the client and the server.
545 +
546 +In most cases though, the configurations on the client and the main
547 +remote will not have changed between 2 fetches or between the initial
548 +clone and a subsequent fetch. This means that the result of a new
549 +protocol negotiation will be the same as the previous result, so the
550 +new fetch will happen in the same way as the previous clone or fetch,
551 +using, or not using, the same LOP(s) as last time.
552 +
553 +"Backfill" or "lazy" fetch
554 +++++++++++++++++++++++++++
555 +
556 +When there is a backfill fetch, the client doesn't necessarily contact
557 +the main remote first. It will try to fetch from its promisor remotes
558 +in the order they appear in the config file, except that a remote
559 +configured using the `extensions.partialClone` config variable will be
560 +tried last. See
561 +link:partial-clone.html#using-many-promisor-remotes[the partial clone documentation].
562 +
563 +This is not new with this effort. In fact this is how multiple remotes
564 +have already been working for around 5 years.
565 +
566 +When using LOPs, having the main remote configured using
567 +`extensions.partialClone`, so it's tried last, makes sense, as missing
568 +objects should only be large blobs that are on LOPs.
569 +
570 +This means that a protocol negotiation will likely not happen as the
571 +missing objects will be fetched from the LOPs, and then there will be
572 +nothing left to fetch from the main remote.
573 +
574 +To secure that, it could be a good idea for LOPs to require a token
575 +from the client when it fetches from them. The client could get the
576 +token when performing a protocol negotiation with the main remote (see
577 +section II.6 above).
578 +
579 +V) Future improvements
580 +----------------------
581 +
582 +It is expected that at the beginning using LOPs will be mostly worth
583 +it either in a corporate context where the Git version that clients
584 +use can easily be controlled, or on repos that are infrequently
585 +accessed. (See the "Could the main remote be bogged down by old or
586 +paranoid clients?" section in the FAQ above.)
587 +
588 +Over time, as more and more clients upgrade to a version that
589 +implements the "promisor-remote" protocol v2 capability described
590 +above in section II.6), it will be worth it to use LOPs more widely.
591 +
592 +A lot of improvements may also help using LOPs more widely. Some of
593 +these improvements are part of the scope of this document like the
594 +following:
595 +
596 + - Implementing a "remote-object-info" command in the
597 + `git cat-file --batch` protocol and its variants to allow main
598 + remotes to respond to requests about large blobs without fetching
599 + them. (Eric Ju has started working on this based on previous work
600 + by Calvin Wan.)
601 +
602 + - Creating better cleanup and offload mechanisms for main remotes
603 + and clients to prevent accumulation of large blobs.
604 +
605 + - Developing more sophisticated protocol negotiation capabilities
606 + between clients and servers for handling LOPs, for example adding
607 + a filter-spec (e.g., blob:limit=<size>) or size limit for
608 + filtering when cloning, or adding a token for LOP authentication.
609 +
610 + - Improving security measures for LOP access, particularly around
611 + token handling and authentication.
612 +
613 + - Developing standardized ways to configure and manage multiple LOPs
614 + across different environments. Especially in the case where
615 + different LOPs serve the same content to clients in different
616 + geographical locations, there is a need for replication or
617 + synchronization between LOPs.
618 +
619 +Some improvements, including some that have been mentioned in the "0)
620 +Non Goals" section of this document, are out of the scope of this
621 +document:
622 +
623 + - Implementing a new object representation for large blobs on the
624 + client side.
625 +
626 + - Developing pluggable ODBs or other object database backends that
627 + could chunk large blobs, dedup the chunks and store them
628 + efficiently.
629 +
630 + - Optimizing data transfer between LOPs and clients/servers,
631 + particularly for incompressible and non-deltifying content.
632 +
633 + - Creating improved client side tools for managing large objects
634 + more effectively, for example tools for migrating from Git LFS or
635 + git-annex, or tools to find which objects could be offloaded and
636 + how much disk space could be reclaimed by offloading them.
637 +
638 +Some improvements could be seen as part of the scope of this document,
639 +but might already have their own separate projects from the Git
640 +project, like:
641 +
642 + - Improving existing remote helpers to access object storage or
643 + developing new ones.
644 +
645 + - Improving existing object storage solutions or developing new
646 + ones.
647 +
648 +Even though all the above improvements may help, this document and the
649 +LOP effort should try to focus, at least first, on a relatively small
650 +number of improvements mostly those that are in its current scope.
651 +
652 +For example introducing pluggable ODBs and a new object database
653 +backend is likely a multi-year effort on its own that can happen
654 +separately in parallel. It has different technical requirements,
655 +touches other part of the Git code base and should have its own design
656 +document(s).