[rt-users] Speeding up RT3

[Matthew Watson]

Following up on this,

I've moved our RT install onto a new Dual 3.2ghz Xeon system with 2gb
ram.

Not really seeing huge improvements.

Still seeing load times of about 5 seconds for index.html , for a user
with
Access to about 20 queues. And 2-4 seconds for a ticket with a single
transaction.



I've tried using Devel::DProf, but the results just don't look correct
(perhaps I'm using it wrong). For a start its getting lots of unstacked
calls, but that seems to be normal.

But User+System Time seems way off. The following is from running
Devel::Dprof with the standalone, and spending a couple of minutes
clicking
Around the UI, loading various tickets, etc.

[root at stratus bin]# time perl -d:DProf standalone_httpd
....
real    1m52.397s
user    0m16.495s
sys     0m1.511s


[root at stratus bin]# dprofpp -O 10 tmon.out
Storable::_freeze has 1 unstacked calls in outer
Exporter::Heavy::heavy_require_version has 1 unstacked calls in outer
CGI::path_info has 1 unstacked calls in outer
CGI::cache has 1 unstacked calls in outer
Net::LDAP::import has -1 unstacked calls in outer
Sys::Syslog::AUTOLOAD has -7 unstacked calls in outer
utf8::AUTOLOAD has -1 unstacked calls in outer
RT::Interface::Web::Handler::new has -1 unstacked calls in outer
Exporter::export_tags has -1 unstacked calls in outer
Net::LDAP::Constant::import has 1 unstacked calls in outer
File::Glob::GLOB_TILDE has 1 unstacked calls in outer
AutoLoader::AUTOLOAD has -3 unstacked calls in outer
Storable::nfreeze has 1 unstacked calls in outer
Exporter::Heavy::heavy_export_ok_tags has 4 unstacked calls in outer
CGI::AUTOLOAD has -6 unstacked calls in outer
utf8::SWASHNEW has 1 unstacked calls in outer
Storable::thaw has 1 unstacked calls in outer
POSIX::AUTOLOAD has -1 unstacked calls in outer
Sys::Syslog::__ANON__ has 7 unstacked calls in outer
Exporter::export_ok_tags has -4 unstacked calls in outer
Exporter::Heavy::heavy_export_to_level has 2 unstacked calls in outer
Exporter::export_to_level has -2 unstacked calls in outer
CGI::request_method has 1 unstacked calls in outer
Exporter::Heavy::heavy_export has 40 unstacked calls in outer
Socket::AUTOLOAD has -10 unstacked calls in outer
CGI::unescapeHTML has 1 unstacked calls in outer
File::Glob::GLOB_BRACE has 1 unstacked calls in outer
Exporter::Heavy::heavy_export_tags has 1 unstacked calls in outer
File::Glob::GLOB_NOMAGIC has 1 unstacked calls in outer
File::Glob::AUTOLOAD has -5 unstacked calls in outer
RT::Interface::Web::Handler::NewCGIHandler has 1 unstacked calls in
outer
File::Glob::GLOB_QUOTE has 1 unstacked calls in outer
CGI::delete has 1 unstacked calls in outer
Fcntl::AUTOLOAD has -8 unstacked calls in outer
Socket::__ANON__ has 10 unstacked calls in outer
POSIX::load_imports has 1 unstacked calls in outer
Fcntl::__ANON__ has 8 unstacked calls in outer
File::Glob::GLOB_ALPHASORT has 1 unstacked calls in outer
Exporter::require_version has -1 unstacked calls in outer
Exporter::export has -40 unstacked calls in outer
CGI::header has 1 unstacked calls in outer
Total Elapsed Time = 103.1376 Seconds
  User+System Time = 8.597692 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
 5.62   0.483  2.200    892   0.0005 0.0025  RT::Tickets::_parser
 5.20   0.447  0.447   1670   0.0003 0.0003  DBD::Oracle::st::_prepare
 4.37   0.376 10.089   3429   0.0001 0.0029
HTML::Mason::Commands::__ANON__
 4.13   0.355  1.717  13121   0.0000 0.0001  RT::Record::LoadByCols
 3.84   0.330  0.464   1457   0.0002 0.0003  Sys::Syslog::connect_udp
 2.87   0.247  0.660  13760   0.0000 0.000
DBIx::SearchBuilder::Record::Cachable::new
 2.45   0.211  0.211 112307   0.0000 0.0000
HTML::Mason::MethodMaker::__ANON__
 2.12   0.182  0.244   6132   0.0000 0.0000  Regexp::Common::_decache
 2.05   0.176  0.424   5828   0.0000 0.0001  DBIx::SearchBuilder::Limit
 2.01   0.173  0.566  13121   0.0000 0.0000
DBIx::SearchBuilder::Record::Cachable::_lookup_primary_RecordCache_key


Any more suggestions on how we can speed RT3 up?

Regards,
Matt.


> -----Original Message-----
> From: Jesse Vincent [mailto:jesse at bestpractical.com]
> Sent: Sunday, October 03, 2004 12:25 AM
> To: Matthew Watson
> Cc: rt-users at lists.fsck.com
> Subject: Re: [rt-users] Speeding up RT3
> 
> >>>
> >>
> >> What DBIx::SearchBuilder are you running?
> >
> > 1.01
> >
> Go up to the latest. We saw noticable performance improvements using
> the latest DBIx::SearchBuilder
> 
> >> How many queues do
> >> non-superusers have on the front page?
> >
> > 18 queues, which have the right "SeeQueue","CreatTicket" and
> > "ReplyToTicket" assigned to "Everyone".
> >
> >
> >
> >>>
> >>> Tracking this down, it's the QuickSearch box that is the major
> > slowdown,
> >>> taking about 6 seconds to load, seeming most of that time is taken
> > up
> >>> processing the ACL in perl.
> >>>
> >>
> >> Is that a guess or do you have profiling results?
> >
> > Profiling using MasonX::Profiler.
> >
> *nod* That gets us down to mason component, but won't tell us anything
> about the underlying libraries ;)
> 
> >> If you don't have profiling results, I'd strongly recommend you use
> >> standalone_httpd and Devel::DProf to profile and post those
results.
> >
> > I'll have a look into doing that.
> 
> Wonderful.
> 
> >
> >> Also, what sorts of CPUs are sitting behind that oracle database?
Have
> >> your DBAs spent time profiling what RT's doing on the database side
> > and
> >> looking for index improvements?
> >
> > I'm not what the CPUs are off hand, it's dual processor sun hardware
> > with 4gb of ram, but I can't remember exactly what, however we have
> > done
> > quite a bit of work making sure the database isn't the cause of the
> > problems, all the queries for the front page take less than 1 second
to
> > run (combined).
> 
> 
> Now _that's's_ interesting. Have you ended up reindexing the Oracle
> database? If so, I know some folks who would love to see the work
> you've done. But that points to library issues. And the SearchBuilder
> issues
> were the ones that were most visible in our profiling runs here at
Best
> Practical. (These days, much of RT's time is spent in mason rendering
> the page itself. Amazon.com has been doing a bunch of mason
performance
> work that will significantly improve that in the next big release)
> 
> Jesse


This email and any files transmitted with it are confidential and intended solely for the 
use of the individual or entity to whom they are addressed. Please notify the sender 
immediately by email if you have received this email by mistake and delete this email 
from your system. Please note that any views or opinions presented in this email are solely
 those of the author and do not necessarily represent those of the organisation. 
Finally, the recipient should check this email and any attachments for the presence of 
viruses. The organisation accepts no liability for any damage caused by any virus 
transmitted by this email.