[rt-users] RT slowness and 500 Internal Server Errors

[Andrew Puschak]

Hello Everyone!

I inherited an RT instance as an administrator. Previous admins who setup and maintained RT have left and we now need to move the database as well as upgrade and fix issues. I'm told Jesse from best practical came here and did the last upgrade and that we have a small support contract with BP but I have not gotten the information yet. After almost a year of administering accounts, queues, troubleshooting, and finding time to attempt an upgrade, I've finally successfully completed a database move and upgrade from 3.8.4 to 4.0.1 with a copy of our production database.

Our running production system is almost out of disk space. It actually ran out of space and I cleaned up the mysqld-slow.log and gained us a about a month. The front end is in one VM and the database in another. I'm told the original plan was to build a dedicated database server and move the database there, but its been stuck in VM's for 2 years now. I have the dedicated server setup and working with a copy of production now and planning to move it over and upgrade.

Problems: Interacting in the web interface is slow. It has been painful for everyone in the company, but appears to get worst and worst. I have a test query (Status = 'new' OR Status = 'open' OR Status = 'stalled' OR Status = 'resolved' OR Status = 'rejected') that completes in about 40 seconds all the time. However, more frequently, it will just display "Internal Server Error". One department who uses it constantly and keeps track well is reporting it used to take them 1.5 hours earlier this year to complete 50 tickets and now it takes 3 hours because they get the error when trying to work. So far the upgraded copy does not timeout but it takes even longer to complete the query, about 1 minute. A nice feature is that the result page loads after about 20 seconds, but doesnt finish loading and populate results till after 1 minute total.

I've read about using AdminCC too prolifically which can cause significant slowdown. We do have a lot of people who are AdminCC (receiving email notifications on every ticket in a queue) in many queues. I will test removing these in my upgraded copy and if thats the case it will be difficult to convince the company that we need to take people off it, most departments rely on email to interact with RT as well as the web interface and need to get every email :(

I'm going to keep reading the mailing list emails I have stored up and keep looking at RT's Wiki as well as the web, but I was wondering if anyone could point me in a direction to look at where our issues are, and what I can try on this upgraded copy I have? Any recommendations for our situation and my plan to move the database, upgrade from 3.8.4 to 4.0.1, as well as resolve our slowness and timeout errors would be greatly appreciated!

Andrew



Andrew Puschak
Senior Unix Administrator
Wolfram Research, Inc.
apuschak at wolfram.com x3329