[svk-users] SVK smerge memory leak?

[Graham Klyne]

Hi,

I'm not sure if this is the right forum for SVK feedback, but here goes...


== Background ==

I'm using SVK to migrate an active development SVN repository from a home-based 
host to a commercial hosted SVN service.  (The reason for using SVK for this is 
that it provides a way to migrate users gradually rather than having to freeze 
activities by a distributed team in the current repository while migrating.)

The current repo is about 1Gb and about 2K revisions.

I'm using SVK-v2.0.99_991, installed from the Dec 2007 development tarball at 
CPAN (http://search.cpan.org/dist/SVK/)


== Observations ==

Grabbing the current repository into an SVK "staging" depot using a combination 
of svnadmin dump files and SVK mirroring has worked well.

Now I'm loading the SVK repository to the hosted service using something like:

  svk mirror /wb/Hosted https://...@svn2.hosted-projects.com/...
  svk sync /wb/Hosted
  svk smerge --incremental --baseless /wb/Thirtover /wb/Hosted

All starts well, maybe a little slow (my upload speed is only about 350Kbps) so 
I leave it running overnight.  Coming back in the morning, the system seems to 
be stuck around revision 950, after running successfully for about 12 hours.  No 
new commits can be seen in the target repository for several hours.

Using 'top', I see two perl processes are hogging all of the systems memory (top 
shows about 50% memory use by each process), and nothing has been committed for 
several hours.  Unfortunately, I didn't make proper note of the process list here :(

The problem is easily enough fixed by:
* aborting the smerge operation
* resyncing the target repository
* restarting the smerge

  svk sync /wb/Hosted
  svk smerge --incremental --baseless /wb/Thirtover /wb/Hosted

(Hmmm... I forgot to drop the --baseless option when restarting, but SVK appears 
to be doing the right things anyway.)

Watching 'top' again, I see the svk process:

[[
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
  ...
root     26353  5.5 13.5  80968 70024 pts/1    S+   09:10   1:37 /usr/bin/perl 
-w /usr/bin/svk smerge --incremental --baseless /wb/Thirtover /wb/Hosted
]]

The memory usage seems once again to be creeping up over time.

Occasionally, I'm seeing a pair of processes:
[[
[root at webbricksystems-svn svk]# ps aux | grep svk
root     26353  5.4 13.9  82864 71752 pts/1    S+   09:10   1:43 /usr/bin/perl 
-w /usr/bin/svk smerge --incremental --baseless /wb/Thirtover /wb/Hosted
root     26574  3.2 13.4  84284 69200 pts/1    S+   09:41   0:00 /usr/bin/perl 
-w /usr/bin/svk smerge --incremental --baseless /wb/Thirtover /wb/Hosted
root     26578  0.0  0.1   3892   648 pts/2    R+   09:41   0:00 grep svk
]]

After 40 minutes of running the new smerge, it's back up to about 17% memory 
utilization according to ps/top, and appears to be growing at an even rate.

Later ... after another 900 revisions have been transferred:

The 50% mentioned earlier appears to be a bit of a red herring - I'm now seeing 
the total system memory use is "overflowing" into the swap area.  I notice that 
the SVK process is periodically creating a second process, which terminates when 
the transaction in progress is done (I guess that's a 'fork' so isn't really 
using 100% memory).

[[
top - 13:32:41 up 1 day,  4:08,  4 users,  load average: 0.30, 0.23, 0.22
Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.6%us,  1.0%sy,  0.0%ni, 97.0%id,  0.0%wa,  0.3%hi,  0.0%si,  0.0%st
Mem:    515524k total,   431472k used,    84052k free,     4456k buffers
Swap:  1048568k total,   226820k used,   821748k free,    84480k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
26353 root      15   0  384m 232m 5420 S  1.7 46.2  15:00.13 svk
]]

#g


-- 
Graham Klyne
Contact info: http://www.ninebynine.org/#Contact