@matheusbsbs / matheus-bagliano / commits / dfa72fdb96

git-svn: reload RA every log-window-size

Despite attempting to use local memory pools everywhere we can, (including our call to SVN::Ra::do_update and all subsequent reporter calls), there does not appear to be a way to force the Git::SVN::Fetcher callbacks to use a pool other than the per-SVN::Ra pool. Git::SVN::Fetcher ends up using the main RA pool which grows monotonically in size for the lifetime of the RA object. Thus the only way to free that memory appears to be to destroy and recreate the RA connection for at every --log-window-size interval. This reduces memory usage over the course of fetching 10K revisions using a test repository created with the script at the end of this commit message. As reported by time(1) on my x86-64 system: before: 54024k after: 28680k Unfortunately, there remains some yet-to-be-tracked-down slow memory growth which would be evident as the `nr' parameter increases in the repository generation script: -----------------------------8<------------------------------ set -e tmp=$(mktemp -d svntestrepo-XXXXXXXX) svnadmin create "$tmp" repo=file://"$(cd $tmp && pwd)" svn co "$repo" "$tmp/wd" cd "$tmp/wd" if ! test -f a then > a svn add a svn commit -m 'A' fi nr=10000 while test $nr -gt 0 do echo $nr > a svn commit -q -m A nr=$((nr - 1)) done echo "repository created in $repo" -----------------------------8<------------------------------ Signed-off-by: Eric Wong <normalperson@yhbt.net>

Eric Wong committed Oct 24, 2014 at 22:53 UTC dfa72fdb96befbd790f623bb2909a347176753c2

1 file changed +13 -8

perl/Git/SVN/Ra.pm

+13 -8

@@ -376,10 +376,19 @@ sub longest_common_path {

376 sub gs_fetch_loop_common {

377 my ($self, $base, $head, $gsv, $globs) = @_;

378 return if ($base > $head);

379 + my $gpool = SVN::Pool->new_default;

380 + my $ra_url = $self->url;

381 + my $reload_ra = sub {

382 + $_[0] = undef;

383 + $self = undef;

384 + $RA = undef;

385 + $gpool->clear;

386 + $self = Git::SVN::Ra->new($ra_url);

387 + $ra_invalid = undef;

388 + };

389 my $inc = $_log_window_size;

390 my ($min, $max) = ($base, $head < $base + $inc ? $head : $base + $inc);

391 my $longest_path = longest_common_path($gsv, $globs);

382 - my $ra_url = $self->url;

392 my $find_trailing_edge;

393 while (1) {

394 my %revs;

@@ -449,13 +458,7 @@ sub gs_fetch_loop_common {

458 "$g->{t}-maxRev";

459 Git::SVN::tmp_config($k, $r);

460 }

452 - if ($ra_invalid) {

453 - $_[0] = undef;

454 - $self = undef;

455 - $RA = undef;

456 - $self = Git::SVN::Ra->new($ra_url);

457 - $ra_invalid = undef;

458 - }

461 + $reload_ra->() if $ra_invalid;

462 }

463 # pre-fill the .rev_db since it'll eventually get filled in

464 # with '0' x40 if something new gets committed

@@ -472,6 +475,8 @@ sub gs_fetch_loop_common {

475 $min = $max + 1;

476 $max += $inc;

477 $max = $head if ($max > $head);

478 +

479 + $reload_ra->();

480 }

481 Git::SVN::gc();

482 }