[rt-users] RT 3.8.8, Apache, and FastCGI memory leak?

[Joshua West]

Hey folks,

We've currently upgraded to RT 3.8.8 on RHEL 5 64-bit.  We've deployed
with Apache 2.2.3 (RHEL supplied), mod_fastcgi 2.4.6.

Unfortunately, about 1-2x a day, the mason_handler.fcgi processes will
progressively grow in memory consumption from running at about 90MB each
to 1-3GB each, and continue to rise, until the machine fills swap and
OOM killer has its fun.  For example, right now you can see some of the
processes which have for some reason, consumed and held onto quite a bit
of memory:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     11216  0.0  0.1 198560  6332 ?        Ss   Jul20   0:00
/usr/sbin/httpd
apache   11218  0.0  0.0 197968  3468 ?        S    Jul20   0:00  \_
/usr/sbin/fcgi-
apache   11219  0.3 29.2 1354192 1227444 ?     S    Jul20   3:36  |   \_
/usr/bin/perl /opt/rt3/bin/mason_handler.fcgi
apache   11229  0.3 29.3 1356172 1229444 ?     S    Jul20   3:46  |   \_
/usr/bin/perl /opt/rt3/bin/mason_handler.fcgi
apache   11232  0.2  7.7 452176 325296 ?       S    Jul20   3:06  |   \_
/usr/bin/perl /opt/rt3/bin/mason_handler.fcgi
apache   11236  0.2  3.1 257672 131180 ?       S    Jul20   2:21  |   \_
/usr/bin/perl /opt/rt3/bin/mason_handler.fcgi

RT is configured for use with mod_fastcgi via the directive:

  FastCgiServer /opt/rt3/bin/mason_handler.fcgi -idle-timeout 120
-processes 4

within Apache's config.

To work around this issue, I've investigated attempting to adjust
mod_fastcgi via the FastCgiConfig directives, so the mason_handler.fcgi
processes would be recycled (spawned, killed, etc) based on demand.  In
theory, that should just start to clean up used memory resources when
there's a lull in activity.  Unfortunately, I haven't had any success in
this.  Seems that the FastCgiConfig directives (singleThreshold,
multiThreshold, minProcesses, maxProcesses, etc.) aren't being honored.

It looks like I can also configure RT to just use mason_handler as an
external FastCGI server, and have Apache/RT communicate via a local UNIX
socket, but I don't think this will solve our issue as the external
server (/opt/rt3/bin/fastcgi_server) doesn't have any advanced process
scheduling abilities like those I was attempting to make use of with
FastCgiConfig directives.

Lastly, this issue seems to be quite similar to one posted at
http://issues.bestpractical.com/Ticket/Display.html?id=15108 -- perhaps
this is more widespread than on just my deployment?

Anyways, any help/input would be greatly appreciated.  I'd rather not
have to cron restart Apache twice a day just to keep the service online.

Thanks!

-- 
Joshua West
Senior Systems Engineer
Brandeis University
http://www.brandeis.edu