[rt-users] Re: [Rt-devel] Performance issues with RT3 (all versions)

Tue Aug 3 15:42:09 EDT 2004

Guys, a lot of words, but no data.
For example DBI profile or perl profile.
MySQL slow log.

Leon wrote:
> I have also had major issues with performance.  We have a quad Xeon with 
> a 1 gig ram, and 15000RPM SCSI drives, that we use for development. We 
> have about 2000 tickets on our system and  the attachments table is 
> about 100 to 150  megs already. We are also using FastCGI,Apache MySQL 4 
> , RT 3.2.1.
We have 7GB of data on IDE with one CPU. Apache1+mp1+MySQL4.0.x. And 
ticket loads less then 15 seconds in most cases.

> 
> When I run IE against RT it sometimes take up to a minute before the 
> ticket is shown.
> I have tried Firefox and found that as the info is retrieved fro the DB, 
> it is displayed. The header will sometime appear a bit slow ( about 5 to 
> 10 seconds) and then the rest of the info comes through one reply or 
> comment at a time. Still taking about 30 to 40 seconds. When running  
> top I can see the MasonHandler.fcgi runs at 99% for the duration of the 
> ticket retrieval.
top is not an answer. For example if MySQL can't do task in memory and 
start use tmptables then you can't see big CPU usage by MySQL, but HDD 
activity is high.

> If another session tries to retrieve a ticket at the some time the 
> duration of all the ticket retrievals increase.
> 
> I have applied the Fast CGI and MYSQL config changes in the WIKIs and no 
> real improvements.
Which? I don't remember any MySQL tricks on WiKi.

> 
> Leon
> Jon Masters wrote:
> 
>> Hi all,
>>
>> For quite some time we have been running RT3 to track support issues, 
>> following my recommendation of the software. Although various 
>> configurations have been used, I am currently debugging an instance 
>> running on a Debian testing box with the specification below because 
>> the faster production machine (running Mandrake release 9) cannot be 
>> tinkered with for the moment and performace problems plague both.
>>
>> Performance is...well horrible. Painfully unpleasant and this is not 
>> with the 40,000 tickets quoted in RT documentation. This is with a 
>> database not exceeding about 50-100MB in size and with only a handfull 
>> of tickets in it. In other words there are almost no tickets.
>>
>> I have tried various different configurations based around RT3 
>> (various versions including but not limited to 3.2.0), MySQL 4, and 
>> Apache 1 and 2 running every possible configuration of mod_perl 1,2 
>> and FastCGI. Essentially I have tried every possible choice available.
>>
>> Query time is so painful that the people here are having to resort to 
>> using regular email and just have RT keep a copy for auditing. I do 
>> unfortunately not have time to delve much in to the bowls of RT (I do 
>> embedded Linux kernel hacking and am not usually a perl person) but 
>> have installed various different profiling options in Apache to at 
>> least give a feel for where this is falling over. I have debug traces 
>> of apache and various other output here and am willing to provide 
>> various additional information and test solutions if I can help to 
>> correct this issue.
>>
>> Clearly the testing machine is not the fastest in the world but it is 
>> an order of magnitude too slow and there is something more fundamental 
>> at fault here. It does not look like MySQL is at fault - it looks like 
>> it is SearchBuilder spewing out way too many searches or something. As 
>> I have read on your mailing list archives and on other online lists, 
>> numerous people report problems with attachments - some of these 
>> tickets have large numbers of attachments but not all, and even then 
>> even listing the queue takes nearly all day.
>>
>> I am not trying to rant here but there is clearly something very weird 
>> going on which I would like to help you guys to fix. You never know, 
>> it might even be possible to convince me that perl isn't such a slow 
>> scripting language after all.
>>
>> Cheers,
>>
>> Jon.
>>
>> --- Specification ---
>>
>> Linux kernel 2.6.6 (Not stock, Debian kernel release 2.6.6-1-686) 
>> running on an Intel Celeron 600MHz CPU. This test box is not the 
>> fastest machine in the world but it runs nothing else other than RT 
>> and a few general system daemons. No windowing system cruft. System 
>> has been tuned using sysctls and ulimits adjusted all in vain, IDE DMA 
>> is on, etc. etc. The production machine exhibits the same problems and 
>> is much faster.
>>
>> (So please assume that the base Debian is configured correctly).
>>
>> To the system has been added MySQL version 4.0.18 (Debian 4.0.18-5).
>>
>> I have also installed RT 3.2.0 as well as other versions.
>>
>> I have performed database OPTIMIZE, ANALYZE, etc. etc.
>>
>> etc. etc. etc. etc.
>>
>> --- dprofpp output ---
>>
>> Storable::nfreeze has 1 unstacked calls in outer
>> Storable::_freeze has 1 unstacked calls in outer
>> Exporter::import has -2 unstacked calls in outer
>> RT::Mason::handler has -1 unstacked calls in outer
>> CGI::delete has 1 unstacked calls in outer
>> AutoLoader::AUTOLOAD has -3 unstacked calls in outer
>> Storable::thaw has 1 unstacked calls in outer
>> Exporter::Heavy::heavy_export has 2 unstacked calls in outer
>> Total Elapsed Time = 196.2333 Seconds
>>   User+System Time = 72.21333 Seconds
>> Exclusive Times
>> %Time ExclSec CumulS #Calls sec/call Csec/c  Name
>>  40.4   29.24 29.418   4647   0.0063 0.0063 
>> DBIx::SearchBuilder::Record::__ANO
>>                                              N__
>>  9.20   6.641 71.487   2297   0.0029 0.0311  HTML::Mason::Request::comp
>>  8.09   5.844  5.844  10197   0.0006 0.0006  HTML::Mason::Request::print
>>  6.79   4.902  4.974     52   0.0943 0.0957 
>> RT::Transaction::BriefDescription
>>  6.58   4.753  4.805     41   0.1159 0.1172  
>> RT::Attachment::ContentLength
>>  4.35   3.139  3.178     25   0.1256 0.1271  RT::Attachment::Content
>>  3.52   2.539  2.639     25   0.1016 0.1055  Text::Quoted::extract
>>  2.28   1.644  1.658     41   0.0401 0.0404  RT::Attachment::Headers
>>  1.65   1.195  1.211     53   0.0226 0.0228  RT::Transactions::Next
>>  1.64   1.186  1.182     52   0.0228 0.0227  RT::Record::CreatedAsString
>>  1.52   1.095  1.119     25   0.0438 0.0448  RT::Transaction::IsInbound
>>  1.37   0.986  1.011     42   0.0235 0.0241  RT::Attachments::Next
>>  0.97   0.697  0.695     50   0.0139 0.0139  RT::Transaction::TicketObj
>>  0.87   0.628  0.635     52   0.0121 0.0122  RT::Record::CreatorObj
>>  0.51   0.369  0.367     20   0.0184 0.0184 
>> DBIx::SearchBuilder::Record::AUTOL
>>                                              OAD
>>
>>
>> _______________________________________________
>> Rt-devel mailing list
>> Rt-devel at lists.bestpractical.com
>> http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-devel
>>
>>
> _______________________________________________
> http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users
> 
> Be sure to check out the RT wiki at http://wiki.bestpractical.com