[rt-users] Performance issues with rt3 - old topic new again..

[Dustin Puryear]

I have read through pages and pages of list archives concerning performance
tuning for rt3. Yes, this issue has been covered in depth, but I'm still going
to bring the subject up anyway because it's Christmas and you have to be
charitable. :)

I have a dual PII 333Mhz system with 512MB of RAM dedicated to the task of
running rt3. (Well, at this point rt3 is just a test install.) I am running
rt3 with FreeBSD 5.1 with SMP, Apache 1.3, modperl1, and MySQL 3.23. 17
queues, 2 users with rights, less than 500 tickets.

I am not satisfied with the response times from rt3. Here are some numbers
pulled from my testing:

(Format is: mo day hr min sec year response-time-in-seconds)

Login:

12 13 22 54 31 2003 5.3498
12 13 23 00 09 2003 5.4111
12 13 23 08 41 2003 5.4133
...

Open a typical queue--this one has 30 new/open tickets:

12 13 22 54 38 2003 5.1363
12 13 23 00 16 2003 5.2176
12 13 23 08 48 2003 4.8305
...

Open a ticket with about 15 comments/corresp. in it:

12 13 22 54 57 2003 16.6257
12 13 23 00 35 2003 16.7649
12 13 23 09 06 2003 16.6477
...

16 seconds! That seems excessive. Also, note that the script is pulling the
same ticket each time--the first may be slow, but after that it *should* be
faster, but it's not. (That is, if the problem was with MySQL.)

Here are the various optimizations that I have tried:

1. Use my-large.cnf as the MySQL my.cnf configuration.
2. Try my-medium.cnf.
3. Find a happy medium between those two values.
4. Use modperl.
5. Use FastCGI.
7. Use MySQL 3.23
8. Use MySQL 4.0
9. Use MySQL 4.0, modperl, my-large.cnf.
10. Other combinations.

None of these changes affected the response time for opening an existing
ticket. The average always stayed around 16.5 seconds.

Looking at the processor utilization, it would appear that httpd is consuming
the CPU the most. However, it never maxes out. Rather, it will usually
steadily reach 50% during my testing (which simulates one user accessing the
system at a time).

I have also tried rt3 on an Alpha 600Mhz with 512MB of RAM. I get the same
response times. (Well, the average for the Alpha for opening an existing
ticket is actually a bit longer.)

I would say I need a faster CPU, but then shouldn't httpd/modperl/rt3 be
hitting a higher percentage of CPU usage than 50%?

I am not hitting swap *at all*.

(The following info is grabbed from the Alpha, which is up right now, but the
configuration is the same for the dual PII.)

Here is my current /etc/my.cnf:

[client]
port            = 3306
socket          = /tmp/mysql.sock

[mysqld]
port            = 3306
socket          = /tmp/mysql.sock
skip-locking
set-variable    = key_buffer=256M
set-variable    = max_allowed_packet=1M
set-variable    = table_cache=256
set-variable    = sort_buffer=1M
set-variable    = record_buffer=1M
set-variable    = myisam_sort_buffer_size=64M
set-variable    = thread_cache=8

set-variable    = thread_concurrency=8
#log-bin
#server-id       = 1

[mysqldump]
quick
set-variable    = max_allowed_packet=16M

[mysql]
no-auto-rehash

[isamchk]
set-variable    = key_buffer=128M
set-variable    = sort_buffer=128M
set-variable    = read_buffer=2M
set-variable    = write_buffer=2M

[myisamchk]
set-variable    = key_buffer=128M
set-variable    = sort_buffer=128M
set-variable    = read_buffer=2M
set-variable    = write_buffer=2M

[mysqlhotcopy]
interactive-timeout

Important processes:

last pid:  5347;  load averages:  0.05,  0.13,  0.09                          
                                                    up 0+09:05:50  03:41:36
30 processes:  1 running, 28 sleeping, 1 stopped
CPU states:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 154M Active, 190M Inact, 47M Wired, 408K Cache, 61M Buf, 72M Free
Swap: 1020M Total, 1020M Free

  PID USERNAME PRI NICE   SIZE    RES STATE    TIME   WCPU    CPU COMMAND
  493 www        4    0 47736K 40856K accept  59:38  0.00%  0.00% httpd
  489 www        4    0 47752K 40840K accept  59:01  0.00%  0.00% httpd
  490 www        4    0 47720K 40840K accept  58:35  0.00%  0.00% httpd
  491 www        4    0 48168K 41312K accept  12:33  0.00%  0.00% httpd
 1846 www        4    0 47992K 41136K accept  12:05  0.00%  0.00% httpd
  492 www        4    0 48224K 41368K accept  11:55  0.00%  0.00% httpd
  429 root      96    0 38896K 34848K select   0:20  0.00%  0.00% httpd
 5343 mysql     96    0   264M  9392K select   0:00  0.00%  0.00% mysqld

Some packages:

p5-HTML-Mason-1.24
mysql-server-3.23.58
apache-1.3.29_1
mod_perl-1.28