Hi Max,<br>now i found the time to try it out.<br><br>Btw: i'm under RT 3.6.5 with RTx-Shredder, but there the Logger Entry is also set. Here the Results first from my test box (only RT running on it, no users)<br><br>
Empty DB<br><br>created 1000 tickets with rt-filler scrip<br><br>1. Tickets shreddered without any index set -> 10 ticket shreddered by default<br><br>time ./rtx-shredder --force --plugin 'Tickets=status,new'<br>
SQL dump file is '/opt/rt3tra/local/sbin/20091124T114147-0001.sql'<br><br>real 0m29.477s<br>user 0m6.638s<br>sys 0m0.323s<br><br><br>2. Ticket shreddered with the following indexes set:<br><br><br> CREATE INDEX SHREDDER_CGM1 ON CachedGroupMembers(MemberId, GroupId, Disabled);<br>
CREATE INDEX SHREDDER_CGM2 ON CachedGroupMembers(ImmediateParentId, MemberId);<br><br> CREATE UNIQUE INDEX SHREDDER_GM1 ON GroupMembers(MemberId, GroupId);<br><br> CREATE INDEX SHREDDER_TXN1 ON Transactions(ReferenceType, OldReference);<br>
CREATE INDEX SHREDDER_TXN2 ON Transactions(ReferenceType, NewReference);<br> CREATE INDEX SHREDDER_TXN3 ON Transactions(Type, OldValue);<br> CREATE INDEX SHREDDER_TXN4 ON Transactions(Type, NewValue);<br>
<br> <br> <br>mysql> CREATE INDEX SHREDDER_CGM1 ON CachedGroupMembers(MemberId, GroupId, Disabled);<br>Query OK, 18411 rows affected (1.23 sec)<br>Records: 18411 Duplicates: 0 Warnings: 0<br><br>mysql> CREATE INDEX SHREDDER_CGM2 ON CachedGroupMembers(ImmediateParentId, MemberId);<br>
Query OK, 18411 rows affected (1.45 sec)<br>Records: 18411 Duplicates: 0 Warnings: 0<br><br>mysql> CREATE UNIQUE INDEX SHREDDER_GM1 ON GroupMembers(MemberId, GroupId);<br>Query OK, 6902 rows affected (0.42 sec)<br>Records: 6902 Duplicates: 0 Warnings: 0<br>
<br>mysql> CREATE INDEX SHREDDER_TXN1 ON Transactions(ReferenceType, OldReference);<br>Query OK, 9940 rows affected (0.78 sec)<br>Records: 9940 Duplicates: 0 Warnings: 0<br><br>mysql> CREATE INDEX SHREDDER_TXN2 ON Transactions(ReferenceType, NewReference);<br>
Query OK, 9940 rows affected (0.91 sec)<br>Records: 9940 Duplicates: 0 Warnings: 0<br><br>mysql> CREATE INDEX SHREDDER_TXN3 ON Transactions(Type, OldValue);<br>Query OK, 9940 rows affected (1.02 sec)<br>Records: 9940 Duplicates: 0 Warnings: 0<br>
<br>mysql> CREATE INDEX SHREDDER_TXN4 ON Transactions(Type, NewValue);<br>Query OK, 9940 rows affected (1.17 sec)<br>Records: 9940 Duplicates: 0 Warnings: 0<br><br>mysql> quit<br>Bye<br>[root@messenger sbin]# time ./rtx-shredder --force --plugin 'Tickets=status,new'<br>
SQL dump file is '/opt/rt3tra/local/sbin/20091124T114403-0001.sql'<br><br>real 0m10.041s<br>user 0m6.612s<br>sys 0m0.354s<br><br> <br>3. Ticket shreddered after removed logger entry in Record.pm<br>
<br>[root@messenger sbin]# time ./rtx-shredder --force --plugin 'Tickets=status,new'<br>SQL dump file is '/opt/rt3tra/local/sbin/20091124T114602-0001.sql'<br><br>real 0m9.475s<br>user 0m6.196s<br>sys 0m0.317s<br>
<br><br>Will try out the same with RT 3.8.6, lets seee what happens<br><br><br>Torsten<br><br><div class="gmail_quote">2009/11/23 Maxwell A. Rathbone <span dir="ltr"><<a href="mailto:mrathbone@sagonet.com">mrathbone@sagonet.com</a>></span><br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div bgcolor="#ffffff" text="#000000">
I noticed a typo in probably the most important line in my message. The
filename is actually:<br>
<br>
/opt/rt3/lib/RT/Shredder/Record.pm<br>
<br>
The line that I suggest to comment out, calls RT's built in Logger()
function that basically just writes information either to the log or to
the screen. <br>
<br>
As with anytime you modify defaults, I make no claims other than what
it had for me. :) I'm actually seeing slightly better than 50%
improvement with that line disabled/commented out. <br>
<br>
I hope others are able to confirm similar experiences. Look forward to
reading about it.<br>
<br>
Max<div><div></div><div class="h5"><br>
<br>
Torsten Brumm wrote:
<blockquote type="cite">Oha, this sounds really useful. Any comment from ruslan if
this is save?<br>
<br>
I have to shred several houndret thousend tickets from 2002-2007 and we
need also around 2 minutes per ticket, will try it out tomorrow!<br>
<br>
Thanks for sharing this<br>
<br>
Torsten<br>
<br>
<div class="gmail_quote">2009/11/23 Maxwell A. Rathbone <span dir="ltr"><<a href="mailto:mrathbone@sagonet.com" target="_blank">mrathbone@sagonet.com</a>></span><br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Hello,<br>
<br>
I'm in the same boat as many others I've seen post. We have 35k tickets<br>
in one of our queues that I'm trying to shred(shame on us for not<br>
automating this previously). I've found the web version of the Shredder<br>
to be god-awful slow. We're talking 10min+ just to shred ONE ticket. So<br>
I discovered the command-line /opt/rt3/sbin/rt-shredder utility. I was<br>
then able to shred ONE ticket in about 5 minutes. I found some<br>
optimization keys to add to the tables, which allowed me to them shred<br>
ONE ticket in about a minute. I then discovered(this really should be in<br>
the documentation!), that if you specify a timeframe with rt-shredder,<br>
you can get MUCH faster processing. I was able to get it down to<br>
21seconds for the shredding of ONE ticket.<br>
<br>
I noticed it was spitting out warning messages each time it deletes<br>
something. I honestly do not care about the output as long as it is<br>
working as expected, so I hunted through the code and was able to<br>
disable the on-screen logging altogether. I'm now able to shred ONE<br>
ticket in about 8-10 seconds.<br>
<br>
For those who are interested in about a 50% reduction in processing time<br>
for the CLI Shredder, edit the file:<br>
/opt/rt3/lib/RT/Shredder/Rercord.pm<br>
<br>
Look for this line:<br>
$RT::Logger->warning( $msg );<br>
<br>
Comment it so it looks like this:<br>
# $RT::Logger->warning( $msg );<br>
<br>
a WORLD of difference from the 10 minutes per ticket I originally was<br>
getting. Now it looks like to shred the 35k might actually take a<br>
palatable amount of time.<br>
<br>
I wanted to share this useful information on the list so it is google<br>
searchable. I'm SURE others will find this helpful.<br>
<br>
BTW, the command I'm using to shred(again, documentation is kinda poor)
is:<br>
./rt-shredder --plugin "Tickets=query,((status = 'deleted' OR status =<br>
'rejected') AND<br>
LastUpdated='2008-10-03');limit,100;with_linked,FALSE;apply_query_to_linked,FALSE"<br>
--force<br>
<br>
Max<br>
_______________________________________________<br>
<a href="http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users" target="_blank">http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users</a><br>
<br>
Community help: <a href="http://wiki.bestpractical.com" target="_blank">http://wiki.bestpractical.com</a><br>
Commercial support: <a href="mailto:sales@bestpractical.com" target="_blank">sales@bestpractical.com</a><br>
<br>
<br>
Discover RT's hidden secrets with RT Essentials from O'Reilly Media.<br>
Buy a copy at <a href="http://rtbook.bestpractical.com" target="_blank">http://rtbook.bestpractical.com</a><br>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
MFG<br>
<br>
Torsten Brumm<br>
<br>
<a href="http://www.brumm.me" target="_blank">http://www.brumm.me</a><br>
<a href="http://www.elektrofeld.de" target="_blank">http://www.elektrofeld.de</a><br>
</blockquote>
<br>
</div></div></div>
<br>_______________________________________________<br>
<a href="http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users" target="_blank">http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users</a><br>
<br>
Community help: <a href="http://wiki.bestpractical.com" target="_blank">http://wiki.bestpractical.com</a><br>
Commercial support: <a href="mailto:sales@bestpractical.com">sales@bestpractical.com</a><br>
<br>
<br>
Discover RT's hidden secrets with RT Essentials from O'Reilly Media.<br>
Buy a copy at <a href="http://rtbook.bestpractical.com" target="_blank">http://rtbook.bestpractical.com</a><br></blockquote></div><br><br clear="all"><br>-- <br>MFG<br><br>Torsten Brumm<br><br><a href="http://www.brumm.me">http://www.brumm.me</a><br>
<a href="http://www.elektrofeld.de">http://www.elektrofeld.de</a><br>