[rt-users] speeding up rt-shredder

Tue Jul 27 13:55:47 EDT 2010

  Did you try turning off logging either through the raising of the 
LogWarn level, or through my suggested modification? I was able to see 
quite a significant difference in simply turning off output from the 
script. It is an inconvenience, but I'm willing to accept that for the 
speed difference.

Also, in general, you are doing an intensive query in your rt-shredder 
command. You may consider breaking this into two separate actions. 
First, marking all of your Subject LIKE '%VIAGRA%' tickets as deleted or 
rejected... and then use rt-shredder to delete the deleted or rejected 
tickets.  It's going to be less intensive on your database server to do 
a Status = 'Rejected' OR Status = 'Deleted' than to do a Subject LIKE 
'%VIAGRA%'.

Personally, I just have my Abuse techs mass-reject the spam we get.. 
Then me, as the admin, just regularly run rt-shredder set to shred 
deleted or rejected tickets. This has allowed us to retain the speed 
using RT & RT:IR. After two years of utilizing RT, I only had this 
realization after I made the post you were linked to. :)

Hope this helps!

Max

On 7/27/2010 1:04 PM, ronald higgins wrote:
> Thanks for the response Maxwell,
>
> So i've added in a date, but alas, still around the 40 second mark.
>
> rt-shredder --force --plugin "Tickets=query,Subject like '%VIAGRA%'
> and LastUpdated>= '2010-07-01 07:00:00' AND LastUpdated<=
> '2010-07-26 07:00:00';limit,2" --sqldump
> /tmp/shredder-restore-tickets.sql
>
> SQL dump file is '/tmp/shredder-restore-tickets.sql'
>
> [Tue Jul 27 17:00:49 2010] [info]: RT::CachedGroupMember-9874009 wiped
> out (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:00:49 2010] [info]: RT::Transaction-13303884 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:00:49 2010] [info]: RT::Group-6080735 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:00:49 2010] [info]: RT::Principal-6080735 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:00:56 2010] [info]: RT::CachedGroupMember-9874011 wiped
> out (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:00:56 2010] [info]: RT::GroupMember-3867641 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:02 2010] [info]: RT::CachedGroupMember-9874006 wiped
> out (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:02 2010] [info]: RT::Transaction-13303881 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:02 2010] [info]: RT::Group-6080732 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:03 2010] [info]: RT::Principal-6080732 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:09 2010] [info]: RT::CachedGroupMember-9878143 wiped
> out (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:09 2010] [info]: RT::GroupMember-3869201 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:16 2010] [info]: RT::CachedGroupMember-9874007 wiped
> out (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:16 2010] [info]: RT::Transaction-13303882 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:16 2010] [info]: RT::Group-6080733 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:16 2010] [info]: RT::Principal-6080733 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:23 2010] [info]: RT::CachedGroupMember-9874008 wiped
> out (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:23 2010] [info]: RT::Transaction-13303883 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:23 2010] [info]: RT::Group-6080734 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:24 2010] [info]: RT::Principal-6080734 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:24 2010] [info]: RT::Attachment-7922989 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:24 2010] [info]: RT::Transaction-13303885 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:25 2010] [info]: RT::Attachment-7922996 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:25 2010] [info]: RT::Transaction-13303897 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:25 2010] [info]: RT::Transaction-13308538 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:25 2010] [info]: RT::Transaction-13308539 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:26 2010] [info]: RT::Attachment-7925504 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:26 2010] [info]: RT::Transaction-13308542 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:26 2010] [info]: RT::Transaction-13309288 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:26 2010] [info]: RT::Ticket-1603953 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:32 2010] [info]: RT::CachedGroupMember-9874027 wiped
> out (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:33 2010] [info]: RT::Transaction-13303900 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:33 2010] [info]: RT::Group-6080747 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:33 2010] [info]: RT::Principal-6080747 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:40 2010] [info]: RT::CachedGroupMember-9874029 wiped
> out (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:40 2010] [info]: RT::GroupMember-3867647 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:46 2010] [info]: RT::CachedGroupMember-9874024 wiped
> out (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:46 2010] [info]: RT::Transaction-13303896 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:46 2010] [info]: RT::Group-6080744 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:46 2010] [info]: RT::Principal-6080744 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:53 2010] [info]: RT::CachedGroupMember-9874028 wiped
> out (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:53 2010] [info]: RT::GroupMember-3867646 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:59 2010] [info]: RT::CachedGroupMember-9874025 wiped
> out (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:59 2010] [info]: RT::Transaction-13303898 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:01:59 2010] [info]: RT::Group-6080745 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:02:00 2010] [info]: RT::Principal-6080745 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:02:06 2010] [info]: RT::CachedGroupMember-9874026 wiped
> out (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:02:06 2010] [info]: RT::Transaction-13303899 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:02:06 2010] [info]: RT::Group-6080746 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:02:07 2010] [info]: RT::Principal-6080746 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:02:07 2010] [info]: RT::Attachment-7922997 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:02:07 2010] [info]: RT::Transaction-13303901 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:02:07 2010] [info]: RT::Attachment-7923000 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:02:07 2010] [info]: RT::Transaction-13303904 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:02:08 2010] [info]: RT::Transaction-13313957 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
> [Tue Jul 27 17:02:08 2010] [info]: RT::Ticket-1603956 wiped out
> (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
>
> Guess i might just have to accept this this will be a lengthy process.
>
>
> On Tue, Jul 27, 2010 at 6:00 PM, Maxwell Rathbone<mrathbone at sagonet.com>  wrote:
>>   What command are you running to run shredder?
>>
>> For me, the best way to speed it up, was to ensure that I was specifying a
>> date range, and to break it down into chunks(like one command per month)
>>
>> Max
>>
>> On 7/27/2010 10:45 AM, ronald higgins wrote:
>>> Thanks for the link, it's helped, but hoping for more gains.
>>>
>>> I've applied the indexes as described and edited the Record.pm and
>>> it's knocked the deletes
>>> down to 40 seconds per ticket. Which translates to about 115 200
>>> records per day or a week solid
>>> to delete my required 700 000 tickets.
>>>
>>> Any other other ideas ?
>>>
>>> Ronald
>>>
>>> On Tue, Jul 27, 2010 at 3:30 PM, G.Booth<G.Booth at lboro.ac.uk>   wrote:
>>>> Hi Ronald
>>>>
>>>> You need to add some more indexes to a few of the tables.
>>>>
>>>> this thread explains all. Think we got to a point where we could shred
>>>> one
>>>> user/ticket every 3-6 secs or so
>>>>
>>>> http://lists.fsck.com/pipermail/rt-users/2009-November/062395.html
>>>>
>>>> hope it helps
>>>>
>>>> regards
>>>> Garry
>>>>
>>>> --
>>>>
>>>> Dr Garry Booth
>>>> IT Services
>>>> Loughborough University
>>>>
>>> Discover RT's hidden secrets with RT Essentials from O'Reilly Media.
>>> Buy a copy at http://rtbook.bestpractical.com
>>
>> Discover RT's hidden secrets with RT Essentials from O'Reilly Media.
>> Buy a copy at http://rtbook.bestpractical.com
>>