[rt-users] Intermittent slow logins on RT 3.6.3

[Allen Chen]

Hi everyone,

We're running RT 3.6.3 on PostgreSQL 8.0.12 and while we've always had 
some intermittent performance issues, it has gotten especially bad in 
the past few weeks.  There are a number of actions that are very slow 
for us, but particularly bad is the amount of time it takes to log in.  
It just took me 136 seconds to log in and display the Home page.  
However, if I refresh the Home page while logged in, it "only" takes 
about 18 seconds.

Here's a mini time line to illustrate a typical login:

    0. Enter username and password and click submit

    1. Immediately afterwards, I see the following in rt.log:
       [Mon Jun 30 21:11:53 2008] [info]: Successful login for allen
    from ... (/var/www/localhost/htdocs/rt/autohandler:256)

    2. 95 seconds later, the login page disappears and the home page
    starts to display

    3. At 136 seconds, the home page is complete

I have set logging in RT to 'debug' so I can see the timing on all the 
queries being executed.  The slowest query generated during that last 
login attempt took 2.50 seconds.  Watching top and iostat on the 
Apache+mod_perl running RT shows a load average around 0.01, CPU 
utilization between 0 and 5%, and disk I/O at <1%.  Watching top and 
iostat on the DB server also shows that it is nowhere near "maxed out."

This problem is intermittent (sometimes it will only take an acceptable 
20 seconds to log in); however, it seems to be slow more often than 
not.  There also seems to be a lot of variance in how slow it gets.  
Sometimes it takes 40 seconds, sometimes over two minutes.

It seems like RT is doing something after authentication and before 
building the home page that makes logins slow.  I have a feeling it 
might be database related, but I cannot find any evidence of any slow 
queries in rt.log.  Could there be any queries that RT executes that 
would not show up in rt.log for some reason?  Also, I have checked 
pg_locks when this is happening but do not see any exclusive locks that 
would cause any queries to wait.  (Note that I /have/ found slow RT 
queries, but not ones associated with logging in.)

Are there any bugs in 3.6.3 that would cause this?  Can anyone suggest 
anything else I should look at or try?

Thanks much,
-Allen

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bestpractical.com/pipermail/rt-users/attachments/20080630/7e7cb239/attachment.htm>