[rt-users] speeding up rt-shredder

Tue Jul 27 13:04:35 EDT 2010

Thanks for the response Maxwell,

So i've added in a date, but alas, still around the 40 second mark.

rt-shredder --force --plugin "Tickets=query,Subject like '%VIAGRA%'
and LastUpdated >= '2010-07-01 07:00:00' AND LastUpdated <=
'2010-07-26 07:00:00';limit,2" --sqldump
/tmp/shredder-restore-tickets.sql

SQL dump file is '/tmp/shredder-restore-tickets.sql'

[Tue Jul 27 17:00:49 2010] [info]: RT::CachedGroupMember-9874009 wiped
out (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:00:49 2010] [info]: RT::Transaction-13303884 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:00:49 2010] [info]: RT::Group-6080735 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:00:49 2010] [info]: RT::Principal-6080735 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:00:56 2010] [info]: RT::CachedGroupMember-9874011 wiped
out (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:00:56 2010] [info]: RT::GroupMember-3867641 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:02 2010] [info]: RT::CachedGroupMember-9874006 wiped
out (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:02 2010] [info]: RT::Transaction-13303881 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:02 2010] [info]: RT::Group-6080732 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:03 2010] [info]: RT::Principal-6080732 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:09 2010] [info]: RT::CachedGroupMember-9878143 wiped
out (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:09 2010] [info]: RT::GroupMember-3869201 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:16 2010] [info]: RT::CachedGroupMember-9874007 wiped
out (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:16 2010] [info]: RT::Transaction-13303882 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:16 2010] [info]: RT::Group-6080733 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:16 2010] [info]: RT::Principal-6080733 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:23 2010] [info]: RT::CachedGroupMember-9874008 wiped
out (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:23 2010] [info]: RT::Transaction-13303883 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:23 2010] [info]: RT::Group-6080734 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:24 2010] [info]: RT::Principal-6080734 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:24 2010] [info]: RT::Attachment-7922989 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:24 2010] [info]: RT::Transaction-13303885 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:25 2010] [info]: RT::Attachment-7922996 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:25 2010] [info]: RT::Transaction-13303897 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:25 2010] [info]: RT::Transaction-13308538 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:25 2010] [info]: RT::Transaction-13308539 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:26 2010] [info]: RT::Attachment-7925504 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:26 2010] [info]: RT::Transaction-13308542 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:26 2010] [info]: RT::Transaction-13309288 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:26 2010] [info]: RT::Ticket-1603953 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:32 2010] [info]: RT::CachedGroupMember-9874027 wiped
out (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:33 2010] [info]: RT::Transaction-13303900 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:33 2010] [info]: RT::Group-6080747 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:33 2010] [info]: RT::Principal-6080747 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:40 2010] [info]: RT::CachedGroupMember-9874029 wiped
out (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:40 2010] [info]: RT::GroupMember-3867647 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:46 2010] [info]: RT::CachedGroupMember-9874024 wiped
out (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:46 2010] [info]: RT::Transaction-13303896 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:46 2010] [info]: RT::Group-6080744 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:46 2010] [info]: RT::Principal-6080744 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:53 2010] [info]: RT::CachedGroupMember-9874028 wiped
out (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:53 2010] [info]: RT::GroupMember-3867646 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:59 2010] [info]: RT::CachedGroupMember-9874025 wiped
out (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:59 2010] [info]: RT::Transaction-13303898 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:01:59 2010] [info]: RT::Group-6080745 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:02:00 2010] [info]: RT::Principal-6080745 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:02:06 2010] [info]: RT::CachedGroupMember-9874026 wiped
out (/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:02:06 2010] [info]: RT::Transaction-13303899 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:02:06 2010] [info]: RT::Group-6080746 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:02:07 2010] [info]: RT::Principal-6080746 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:02:07 2010] [info]: RT::Attachment-7922997 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:02:07 2010] [info]: RT::Transaction-13303901 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:02:07 2010] [info]: RT::Attachment-7923000 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:02:07 2010] [info]: RT::Transaction-13303904 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:02:08 2010] [info]: RT::Transaction-13313957 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)
[Tue Jul 27 17:02:08 2010] [info]: RT::Ticket-1603956 wiped out
(/root/rt-3.8.7/sbin/../lib/RT/Shredder/Record.pm:236)

Guess i might just have to accept this this will be a lengthy process.

On Tue, Jul 27, 2010 at 6:00 PM, Maxwell Rathbone <mrathbone at sagonet.com> wrote:
>  What command are you running to run shredder?
>
> For me, the best way to speed it up, was to ensure that I was specifying a
> date range, and to break it down into chunks(like one command per month)
>
> Max
>
> On 7/27/2010 10:45 AM, ronald higgins wrote:
>>
>> Thanks for the link, it's helped, but hoping for more gains.
>>
>> I've applied the indexes as described and edited the Record.pm and
>> it's knocked the deletes
>> down to 40 seconds per ticket. Which translates to about 115 200
>> records per day or a week solid
>> to delete my required 700 000 tickets.
>>
>> Any other other ideas ?
>>
>> Ronald
>>
>> On Tue, Jul 27, 2010 at 3:30 PM, G.Booth<G.Booth at lboro.ac.uk>  wrote:
>>>
>>> Hi Ronald
>>>
>>> You need to add some more indexes to a few of the tables.
>>>
>>> this thread explains all. Think we got to a point where we could shred
>>> one
>>> user/ticket every 3-6 secs or so
>>>
>>> http://lists.fsck.com/pipermail/rt-users/2009-November/062395.html
>>>
>>> hope it helps
>>>
>>> regards
>>> Garry
>>>
>>> --
>>>
>>> Dr Garry Booth
>>> IT Services
>>> Loughborough University
>>>
>> Discover RT's hidden secrets with RT Essentials from O'Reilly Media.
>> Buy a copy at http://rtbook.bestpractical.com
>
>
> Discover RT's hidden secrets with RT Essentials from O'Reilly Media.
> Buy a copy at http://rtbook.bestpractical.com
>