[Rt-devel] Performance issues with RT3 (all versions)

[Jon Masters]

Hi all,

For quite some time we have been running RT3 to track support issues, 
following my recommendation of the software. Although various 
configurations have been used, I am currently debugging an instance 
running on a Debian testing box with the specification below because the 
faster production machine (running Mandrake release 9) cannot be 
tinkered with for the moment and performace problems plague both.

Performance is...well horrible. Painfully unpleasant and this is not 
with the 40,000 tickets quoted in RT documentation. This is with a 
database not exceeding about 50-100MB in size and with only a handfull 
of tickets in it. In other words there are almost no tickets.

I have tried various different configurations based around RT3 (various 
versions including but not limited to 3.2.0), MySQL 4, and Apache 1 and 
2 running every possible configuration of mod_perl 1,2 and FastCGI. 
Essentially I have tried every possible choice available.

Query time is so painful that the people here are having to resort to 
using regular email and just have RT keep a copy for auditing. I do 
unfortunately not have time to delve much in to the bowls of RT (I do 
embedded Linux kernel hacking and am not usually a perl person) but have 
installed various different profiling options in Apache to at least give 
a feel for where this is falling over. I have debug traces of apache and 
various other output here and am willing to provide various additional 
information and test solutions if I can help to correct this issue.

Clearly the testing machine is not the fastest in the world but it is an 
order of magnitude too slow and there is something more fundamental at 
fault here. It does not look like MySQL is at fault - it looks like it 
is SearchBuilder spewing out way too many searches or something. As I 
have read on your mailing list archives and on other online lists, 
numerous people report problems with attachments - some of these tickets 
have large numbers of attachments but not all, and even then even 
listing the queue takes nearly all day.

I am not trying to rant here but there is clearly something very weird 
going on which I would like to help you guys to fix. You never know, it 
might even be possible to convince me that perl isn't such a slow 
scripting language after all.

Cheers,

Jon.

--- Specification ---

Linux kernel 2.6.6 (Not stock, Debian kernel release 2.6.6-1-686) 
running on an Intel Celeron 600MHz CPU. This test box is not the fastest 
machine in the world but it runs nothing else other than RT and a few 
general system daemons. No windowing system cruft. System has been tuned 
using sysctls and ulimits adjusted all in vain, IDE DMA is on, etc. etc. 
The production machine exhibits the same problems and is much faster.

(So please assume that the base Debian is configured correctly).

To the system has been added MySQL version 4.0.18 (Debian 4.0.18-5).

I have also installed RT 3.2.0 as well as other versions.

I have performed database OPTIMIZE, ANALYZE, etc. etc.

etc. etc. etc. etc.

--- dprofpp output ---

Storable::nfreeze has 1 unstacked calls in outer
Storable::_freeze has 1 unstacked calls in outer
Exporter::import has -2 unstacked calls in outer
RT::Mason::handler has -1 unstacked calls in outer
CGI::delete has 1 unstacked calls in outer
AutoLoader::AUTOLOAD has -3 unstacked calls in outer
Storable::thaw has 1 unstacked calls in outer
Exporter::Heavy::heavy_export has 2 unstacked calls in outer
Total Elapsed Time = 196.2333 Seconds
   User+System Time = 72.21333 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
  40.4   29.24 29.418   4647   0.0063 0.0063 
DBIx::SearchBuilder::Record::__ANO
                                              N__
  9.20   6.641 71.487   2297   0.0029 0.0311  HTML::Mason::Request::comp
  8.09   5.844  5.844  10197   0.0006 0.0006  HTML::Mason::Request::print
  6.79   4.902  4.974     52   0.0943 0.0957 
RT::Transaction::BriefDescription
  6.58   4.753  4.805     41   0.1159 0.1172  RT::Attachment::ContentLength
  4.35   3.139  3.178     25   0.1256 0.1271  RT::Attachment::Content
  3.52   2.539  2.639     25   0.1016 0.1055  Text::Quoted::extract
  2.28   1.644  1.658     41   0.0401 0.0404  RT::Attachment::Headers
  1.65   1.195  1.211     53   0.0226 0.0228  RT::Transactions::Next
  1.64   1.186  1.182     52   0.0228 0.0227  RT::Record::CreatedAsString
  1.52   1.095  1.119     25   0.0438 0.0448  RT::Transaction::IsInbound
  1.37   0.986  1.011     42   0.0235 0.0241  RT::Attachments::Next
  0.97   0.697  0.695     50   0.0139 0.0139  RT::Transaction::TicketObj
  0.87   0.628  0.635     52   0.0121 0.0122  RT::Record::CreatorObj
  0.51   0.369  0.367     20   0.0184 0.0184 
DBIx::SearchBuilder::Record::AUTOL
                                              OAD