[rt-users] Performance with many attachments and transactions

[Nate Duehr]

Ben Goodwin wrote:
> I hate to chime in with a "me too" especially without hard data but:
> 
> We were running 3.5.something on a dual P3 1.4 with 2 gig ram and 15kRPM
> SCSI RAID as the webserver and the same configuration for the database on
> another box.  Loading tickets with any reasonable amount of history to them
> was slow, and tickets with a lot of history would take 90 seconds to load.

This seems really wrong from what I've seen.  Running Apache v1, and 
MySQL - I see huge tickets take less time on an Athlon 2500 with 1G of 
RAM.

Are you sure you weren't struggling with the Linux kernel's general 
problems above 4GB of RAM?  If the kernel wasn't compiled correctly, 
there were some serious performance problems above 4GB.

Were you ever able to classify the latency caused by not having the DB 
and the webserver be the same machine in this setup above?

> All the time was spent in Perl (not the database, which was 85% in RAM).  So
> I decided to upgrade.  We now have the webserver and DB server on a single
> dual P4 Xeon 3.4GHz (2MB cache 800 MHz bus) with 6 gig of ram on RHEL4.  The
> DB is about 1.5 gig so the ENTIRE DB is in RAM - DB calls are nice and fast.

That helps a lot.  I found that using even the example MySQL "large" 
database configuration stuff in the mysql docs also helped performance 
immensely.  Even though bottlenecks LOOKED like they were Perl, Perl was 
waiting on the DB.

Watching vmstat for page faults, since my server also does "other" 
things, I do see page faults when the server's busy -- I should add RAM 
-- but I tweaked a while on mysql settings to try to find a happy medium 
where if three or so users are pulling pages at the same time, the box 
isn't page faulting on the DB.

Later on, I also realized in my little project, that I didn't need old 
tickets so badly that I had to have them online, and I used Ruslan's 
Shredder to dump about three years of DB content not too long ago -- the 
system sped WAY up on things like huge searches that included tickets 
back that old, but it didn't really change the system speed feel for 
single ticket work.

Attachments seem to be the real performance killer.  The more 
attachments, the slower pages render.  Without attachments, or with only 
one or two, I have no complaints -- especially on such cheap hardware!

> Full content searches take < 10 seconds (MySQL).  However, ticket displays,
> while faster, are not nearly as improved as I'd hoped.  The same 90-second
> ticket is down to about 60 seconds - again, all perl (FCGI under Apache2),
> sitting at 100% CPU until the rendering is done.  One of these days I'll
> spend time looking into this but thought I'd at least chime in to indicate
> "you're not alone" as well as indicate that even with very beefy hardware,
> it's still slow.

It sure feels like whenever I use it that Apache 2 is somewhat bloated. 
  You may (if you can) see if you can benchmark the same pages from an 
Apache1 setup in a temporary directory on the same box, if you're not 
averse to using older versions of Apache for some reason.

Nate