[rt-users] Resend today's emails?

[Christopher Short]

Thanks Mark, I haven't had a chance to look at your script yet, but if it does what you promise then it will save much agony! :-)
We're very grateful to you.
regards,
Christopher

-----Original Message-----
From: Mark Chappell [mailto:m.d.chappell at bath.ac.uk]
Sent: Wednesday, 27 February 2008 8:32 PM
To: Christopher Short
Cc: rt-users at lists.bestpractical.com
Subject: Re: [rt-users] Resend today's emails?

Christopher Short wrote:
> Hi,
> every so often our RT 3.6.1 silently stops forwarding emails on to the appropriate recipients, having used up a shocking amount of memory. It's the "Cannot allocate memory" problem, which I gather is a known bug. We're setting up a process to restart RT every night, that will hopefully reduce the occurrence of this problem.

We've had the same thing happen.  There are a couple of tricks you can
play with Apache to reduce the problem.  However, it's also worth noting
that it is possible to resend the mails automatically...

I used the attached perl script to rerun the scrips on the transactions
that failed to send mails.  It's possible to parse out the failed scrips
from the logs

The Message-ID of the failed message will turn up in the apache error
logs.  Then if you search the rt logs (in our case /var/log/messages
because we're sending them through syslog) for the Message-ID of the
message it was trying to send, you'll see a line like

Feb 17 11:44:16 chas RT:
<rt-3.6.1-14703-1203248655-1286.361410-1253-0 at bath.ac.uk>
#361410/1892333 - Scrip 1253
(/usr/share/request-tracker3.6/lib/RT/Action/SendEmail.pm:2
38)

The important bit is "#361410/1892333"  This is the ticket number and
the transaction that caused the scrip to run.

I used a combination of grep, cut, uniq and xargs to tease the
transaction IDs out of the logs and then just re-ran the scrips on those
transactions.

Admittedly this could also reopen or close tickets, but if the user's
not actually seen any reply then they're unlikely to have taken things
further...

Back to Apache...

There are tricks you can play with leaky apache processes...  With a bit
of empirical playing I ended up with the following in the main apache
config.

=====

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_prefork_module>
     StartServers          5
     MinSpareServers       2
     MaxSpareServers       5
     MaxClients          150
     MaxRequestsPerChild 1000
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_worker_module>
     StartServers          2
     MaxClients          150
     MinSpareThreads      25
     MaxSpareThreads      75
     ThreadsPerChild      25
     MaxRequestsPerChild 1000
</IfModule>

======

What it basically does is causes the children to die after answering a
thousand requests, freeing up some of the leaked memory.



Hope this helps,


Mark

--
Mark Chappell
Unix Systems Administrator