[rt-users] Resend today's emails?

[Mark Chappell]

Christopher Short wrote:
> Hi,
> every so often our RT 3.6.1 silently stops forwarding emails on to the appropriate recipients, having used up a shocking amount of memory. It's the "Cannot allocate memory" problem, which I gather is a known bug. We're setting up a process to restart RT every night, that will hopefully reduce the occurrence of this problem.

We've had the same thing happen.  There are a couple of tricks you can 
play with Apache to reduce the problem.  However, it's also worth noting 
that it is possible to resend the mails automatically...

I used the attached perl script to rerun the scrips on the transactions 
that failed to send mails.  It's possible to parse out the failed scrips 
from the logs

The Message-ID of the failed message will turn up in the apache error 
logs.  Then if you search the rt logs (in our case /var/log/messages 
because we're sending them through syslog) for the Message-ID of the 
message it was trying to send, you'll see a line like

Feb 17 11:44:16 chas RT: 
<rt-3.6.1-14703-1203248655-1286.361410-1253-0 at bath.ac.uk> 
#361410/1892333 - Scrip 1253 
(/usr/share/request-tracker3.6/lib/RT/Action/SendEmail.pm:2
38)

The important bit is "#361410/1892333"  This is the ticket number and 
the transaction that caused the scrip to run.

I used a combination of grep, cut, uniq and xargs to tease the 
transaction IDs out of the logs and then just re-ran the scrips on those 
transactions.

Admittedly this could also reopen or close tickets, but if the user's 
not actually seen any reply then they're unlikely to have taken things 
further...

Back to Apache...

There are tricks you can play with leaky apache processes...  With a bit 
of empirical playing I ended up with the following in the main apache 
config.

=====

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_prefork_module>
     StartServers          5
     MinSpareServers       2
     MaxSpareServers       5
     MaxClients          150
     MaxRequestsPerChild 1000
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_worker_module>
     StartServers          2
     MaxClients          150
     MinSpareThreads      25
     MaxSpareThreads      75
     ThreadsPerChild      25
     MaxRequestsPerChild 1000
</IfModule>

======

What it basically does is causes the children to die after answering a 
thousand requests, freeing up some of the leaked memory.



Hope this helps,


Mark

-- 
Mark Chappell
Unix Systems Administrator
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: rt-rerun-scrips
URL: <http://lists.bestpractical.com/pipermail/rt-users/attachments/20080227/05821785/attachment.ksh>