[rt-users] rt-serializer and rt-importer

[Alex Vandiver]

On Tue, 2014-03-11 at 08:51 -0700, Tim Gustafson wrote:
> And one last follow-up about this: here are my import statistics.  The
> importer used up to about 800MB of RAM, so better than the serializer,
> but still pretty high.

Both of those (6G for serializing, and 800M for importing) are
definitely higher than I'd expect.  If you're on 4.2.2, then you have
the fixes to explicitly turn off the caching layer.

If you're interested in poking further, doing some analysis of the
in-memory objects to determine what's leaking would be helpful.
Instrumenting the main loop of either of the serializer or importer with
Devel::Gladiator::arena_ref_counts to look for what sorts of objects are
leaking might be instructive.

> Also, 108 hours seems like a long time to import records that took only
> 4 hours to export, and this is on a pretty high-end MySQL server.  Is
> there any way to speed things up a bit?

Yeesh, that's pretty bad.  I'd try running under Devel::NYTProf to see
where the hotspots are -- I don't have any off-the-cuff suggestions.
 - Alex