[svk-devel] Memory use explosion on large delete

[Daniel Jacobowitz]

Some more information on the problem I reported on IRC this afternoon.

I maintain an SVK mirror of GCC (two actually).  One of the machines
with a mirror ran out of memory last night and barely staggered back to
life, and the other is still down.  Richard Earnshaw experienced
something similar, and found that his happened while svk was mirroring
revision r121126 from svn://gcc.gnu.org/svn/gcc.  This is the deletion
of about 15000 binary files.

SVK sits in a loop, eating CPU, for about three minutes.  Then it
starts reading revisions and writing transactions, but memory usage
goes straight up while it does that - it hit 1.4GB in about thirty
seconds.  I don't know how high up it'll go, but at least past 3.5GB.

I am guessing SVN doesn't really need this much memory for what it's
doing, and there's a pool leak somewhere.  Is there an easy way for
me to get APR to dump the state of various pools?

We use this mirror every day, so having to disable it really hurts :-(

I ran it again with a ulimit set, and this is where it finally crashed:

#0  0x00002aaaab1c107b in raise () from /lib/libc.so.6
#1  0x00002aaaab1c284e in abort () from /lib/libc.so.6
#2  0x00002aaaabac8e89 in svn_pool_create_ex_debug ()
   from /usr/lib/libsvn_subr-1.so.1
#3  0x00002aaaabd020e6 in apr_palloc () from /usr/lib/libapr-1.so.0
#4  0x00002aaaabcff514 in apr_file_open () from /usr/lib/libapr-1.so.0
#5  0x00002aaaabac1cd6 in svn_io_file_open ()
   from /usr/lib/libsvn_subr-1.so.1
#6  0x00002aaaae4b8af3 in svn_fs_fs__get_node_revision ()
   from /usr/lib/libsvn_fs_fs-1.so.1
#7  0x00002aaaae4b0acb in svn_fs_fs__dag_get_fs ()
   from /usr/lib/libsvn_fs_fs-1.so.1
#8  0x00002aaaae4b1266 in svn_fs_fs__dag_dir_entries ()
   from /usr/lib/libsvn_fs_fs-1.so.1
#9  0x00002aaaae4b1316 in svn_fs_fs__dag_open ()
   from /usr/lib/libsvn_fs_fs-1.so.1
#10 0x00002aaaae4be487 in svn_fs_fs__revision_root ()
   from /usr/lib/libsvn_fs_fs-1.so.1
#11 0x00002aaaae4be619 in svn_fs_fs__revision_root ()
   from /usr/lib/libsvn_fs_fs-1.so.1
#12 0x00002aaaae4be90c in svn_fs_fs__node_created_rev ()
   from /usr/lib/libsvn_fs_fs-1.so.1
#13 0x00002aaaae4be99b in svn_fs_fs__check_path ()
   from /usr/lib/libsvn_fs_fs-1.so.1
#14 0x00002aaaae059cd7 in svn_repos_get_commit_editor ()
   from /usr/lib/libsvn_repos-1.so.1
#15 0x00002aaaafcbdc18 in _wrap_svn_delta_editor_invoke_delete_entry ()
   from /usr/lib/perl5/auto/SVN/_Delta/_Delta.so
#16 0x00002aaaaac4eb95 in Perl_pp_entersub () from /usr/lib/libperl.so.5.8
#17 0x00002aaaaac4d36e in Perl_runops_standard ()
   from /usr/lib/libperl.so.5.8
#18 0x00002aaaaac7d19a in Perl_pp_leavetry () from
/usr/lib/libperl.so.5.8
#19 0x00002aaaaac4d36e in Perl_runops_standard ()
   from /usr/lib/libperl.so.5.8
#20 0x00002aaaaabf7ec8 in Perl_call_sv () from /usr/lib/libperl.so.5.8
#21 0x00002aaaabbe74f7 in svn_swig_pl_callback_thunk ()
   from /usr/lib/libsvn_swig_perl-1.so.1
#22 0x00002aaaabbe9d66 in svn_swig_pl_thunk_log_receiver ()
   from /usr/lib/libsvn_swig_perl-1.so.1
#23 0x00002aaaae280372 in ?? () from /usr/lib/libsvn_ra_svn-1.so.1
#24 0x00002aaaadd081c1 in _wrap_svn_ra_get_log ()
   from /usr/lib/perl5/auto/SVN/_Ra/_Ra.so
#25 0x00002aaaaac4eb95 in Perl_pp_entersub () from
/usr/lib/libperl.so.5.8
#26 0x00002aaaaac4d36e in Perl_runops_standard ()
   from /usr/lib/libperl.so.5.8
#27 0x00002aaaaabf89ca in perl_run () from /usr/lib/libperl.so.5.8
#28 0x00000000004017ac in main ()


-- 
Daniel Jacobowitz
CodeSourcery