[rt-users] sendmailpipe returns EX_TEMPFAIL

[Nathan Cutler]

Greetings:

We are having an problem with Postfix sendmail returning "code 75"
(EX_TEMPFAIL) on outgoing emails from RT. I have been monitoring the
bug and can find no rhyme or reason to it. Most of the time, emails
leave RT without any trouble. But occasionally they fail
unreproducibly like this:

Aug 12 08:01:16 perelman RT:
<rt-4.0.13-29527-1376287275-1791.35212-4-0 at suse.de> #35212/333905 -
Scrip 4 On Create Notify AdminCcs
(/usr/lib/perl5/vendor_perl/5.10.0/RT/Action/SendEmail.pm:285)
Aug 12 08:01:16 perelman RT:
<rt-4.0.13-29527-1376287275-1791.35212-4-0 at suse.de>:
`/usr/sbin/sendmail -oi -t` exited with code 75
(/usr/lib/perl5/vendor_perl/5.10.0/RT/Interface/Email.pm:486)
Aug 12 08:01:16 perelman RT:
<rt-4.0.13-29527-1376287275-1791.35212-4-0 at suse.de>: Could not send
mail with command `/usr/sbin/sendmail -oi -t`:
<rt-4.0.13-29527-1376287275-1791.35212-4-0 at suse.de>:
`/usr/sbin/sendmail -oi -t` exited with code 75 at
/usr/lib/perl5/vendor_perl/5.10.0/RT/Interface/Email.pm line 487.
Stack:   [/usr/lib/perl5/vendor_perl/5.10.0/RT/Interface/Email.pm:487]
  [/usr/lib/perl5/vendor_perl/5.10.0/RT/Action/SendEmail.pm:292]
[/usr/lib/perl5/vendor_perl/5.10.0/RT/Action/SendEmail.pm:114]
[/usr/lib/perl5/vendor_perl/5.10.0/RT/ScripAction.pm:232]
[/usr/lib/perl5/vendor_perl/5.10.0/RT/Scrip.pm:475]
[/usr/lib/perl5/vendor_perl/5.10.0/RT/Scrips.pm:188]
[/usr/lib/perl5/vendor_perl/5.10.0/RT/Transaction.pm:201]
[/usr/lib/perl5/vendor_perl/5.10.0/RT/Record.pm:1504]
[/usr/lib/perl5/vendor_perl/5.10.0/RT/Ticket.pm:682]
[/usr/lib/perl5/vendor_perl/5.10.0/RT/Interface/Email.pm:1569]
[/usr/share/request-tracker/html/REST/1.0/NoAuth/mail-gateway:61]
(/usr/lib/perl5/vendor_perl/5.10.0/RT/Interface/Email.pm:491)

This is RT 4.0.13 and we are running it on a virtual machine with 8GB
of RAM. Free RAM is around 80%, so the problem is not memory-related.
There is one other mod_perl application running on the same machine,
but adding "PerlOptions +Parent" to the Apache vhost configuration
didn't help. (Do I need to add it to the vhost configuration of the
other mod_perl app, too?)

In RT_SiteConfig.pm I have:

Set($SendmailCommand, 'sendmailpipe');
Set($SendmailArguments, '-oi -t');

I'm running a fairly modern Postfix (v. 2.5.13) on a fairly modern OS
(SLES11-SP2). Here's apache2ctl -V:

# apache2ctl -V
Server version: Apache/2.2.12 (Linux/SUSE)
Server built:   Mar 27 2013 18:47:49
Server's Module Magic Number: 20051115:23
Server loaded:  APR 1.3.3, APR-Util 1.3.4
Compiled using: APR 1.3.3, APR-Util 1.3.4
Architecture:   64-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/srv/www"
 -D SUEXEC_BIN="/usr/sbin/suexec2"
 -D DEFAULT_PIDLOG="/var/run/httpd2.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="/var/run/accept.lock"
 -D DEFAULT_ERRORLOG="/var/log/apache2/error_log"
 -D AP_TYPES_CONFIG_FILE="/etc/apache2/mime.types"
 -D SERVER_CONFIG_FILE="/etc/apache2/httpd.conf"

What appears to be happening is this: for reasons unknown,
occasionally when RT goes to send an email, the sendmailpipe command
('/usr/bin/sendmail -oi -t') gets executed but nothing is piped in.
This causes sendmail to report:

postfix/sendmail[23352]: fatal: wwwrun(30): No recipient addresses
found in message header

and RT logs the "code 75" error quoted above. In the web UI, when I
look at the transaction "with headers", it says:

X-RT-Interface: API
MIME-Version: 1.0
X-Mailer: MIME-tools 5.502 (Entity 5.502)
Content-Type: text/plain; charset="utf-8"
Content-Disposition: inline
Content-Transfer-Encoding: binary
X-RT-Original-Encoding: ascii
RT-Message-ID: <rt-4.0.13-29527-1376293428-636.35212-0-0 at suse.de>
Content-Length: 105

Sending the previous mail has failed.  Please contact your admin, they
can find more details in the logs.

Yesterday I added the '-v' switch to the sendmail command line, which
is causing mail delivery reports to be sent to a local user on the
machine. These mail delivery reports reproduce all the headers. I have
not managed to "capture" the bug in this way, yet, however. But as
soon as I do I will post the mail delivery report here.

Does anyone have any other ideas for debugging this issue? Especially
I am interested in how I could confirm or deny that it's related to
mod_perl "cohabitation" -- i.e. two different Perl applications in a
single mod_perl host?

Nathan