[rt-users] Speeding up CLI RT::Shredder

[Maxwell A. Rathbone]

I noticed a typo in probably the most important line in my message. The 
filename is actually:

/opt/rt3/lib/RT/Shredder/Record.pm

The line that I suggest to comment out, calls RT's built in Logger() 
function that basically just writes information either to the log or to 
the screen.

As with anytime you modify defaults, I make no claims other than what it 
had for me. :) I'm actually seeing slightly better than 50% improvement 
with that line disabled/commented out.

I hope others are able to confirm similar experiences. Look forward to 
reading about it.

Max

Torsten Brumm wrote:
> Oha, this sounds really useful. Any comment from ruslan if this is save?
>
> I have to shred several houndret thousend tickets from 2002-2007 and 
> we need also around 2 minutes per ticket, will try it out tomorrow!
>
> Thanks for sharing this
>
> Torsten
>
> 2009/11/23 Maxwell A. Rathbone <mrathbone at sagonet.com 
> <mailto:mrathbone at sagonet.com>>
>
>     Hello,
>
>     I'm in the same boat as many others I've seen post. We have 35k
>     tickets
>     in one of our queues that I'm trying to shred(shame on us for not
>     automating this previously). I've found the web version of the
>     Shredder
>     to be god-awful slow. We're talking 10min+ just to shred ONE
>     ticket. So
>     I discovered the command-line /opt/rt3/sbin/rt-shredder utility. I was
>     then able to shred ONE ticket in about 5 minutes. I found some
>     optimization keys to add to the tables, which allowed me to them shred
>     ONE ticket in about a minute. I then discovered(this really should
>     be in
>     the documentation!), that if you specify a timeframe with rt-shredder,
>     you can get MUCH faster processing. I was able to get it down to
>     21seconds for the shredding of ONE ticket.
>
>     I noticed it was spitting out warning messages each time it deletes
>     something. I honestly do not care about the output as long as it is
>     working as expected, so I hunted through the code and was able to
>     disable the on-screen logging altogether. I'm now able to shred ONE
>     ticket in about 8-10 seconds.
>
>     For those who are interested in about a 50% reduction in
>     processing time
>     for the CLI Shredder, edit the file:
>     /opt/rt3/lib/RT/Shredder/Rercord.pm
>
>     Look for this line:
>        $RT::Logger->warning( $msg );
>
>     Comment it so it looks like this:
>     #    $RT::Logger->warning( $msg );
>
>     a WORLD of difference from the 10 minutes per ticket I originally was
>     getting. Now it looks like to shred the 35k might actually take a
>     palatable amount of time.
>
>     I wanted to share this useful information on the list so it is google
>     searchable. I'm SURE others will find this helpful.
>
>     BTW, the command I'm using to shred(again, documentation is kinda
>     poor) is:
>     ./rt-shredder --plugin "Tickets=query,((status = 'deleted' OR status =
>     'rejected') AND
>     LastUpdated='2008-10-03');limit,100;with_linked,FALSE;apply_query_to_linked,FALSE"
>     --force
>
>     Max
>     _______________________________________________
>     http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users
>
>     Community help: http://wiki.bestpractical.com
>     Commercial support: sales at bestpractical.com
>     <mailto:sales at bestpractical.com>
>
>
>     Discover RT's hidden secrets with RT Essentials from O'Reilly Media.
>     Buy a copy at http://rtbook.bestpractical.com
>
>
>
>
> -- 
> MFG
>
> Torsten Brumm
>
> http://www.brumm.me
> http://www.elektrofeld.de

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bestpractical.com/pipermail/rt-users/attachments/20091123/ccf9c56b/attachment.htm>