[rt-users] apache httpd segfaults

[Patrick Hess]

I don't have any RHEL5.x servers at my disposal but, based on what I know
about perl and RHEL4, from your strace output it looks like your perl
installation may be questionable or at fault here.    I recommend verifying
your perl-5.8.8 RPM.  Don't be surprised if files show up as "missing".
When you install updated perl modules, the install may choose to uninstall
the old versions.  If there are "missing" files from the RPM, check in
"vendor_perl" and "site_perl" to make sure they're installed elsewhere.

I have experienced similar segfaults from Apache-2.0 when a server is short
on memory because I have set vm.overcommit_memory=2 and
vm.overcommit_ratio=100 on my RHEL4 servers.  This essentially puts a limit
on how much memory the kernel can claim to have and thereby forces malloc()
to return NULL if the server doesn't have enough free memory to allocate.
It seems that Apache-2.0 doesn't know how to deal with the inability to
allocate memory; it's possible that this bug is still present in Apache-2.2.

Have you attempted to allow Apache to leave coredumps?  In Apache-2.0 this
was done by setting the "CoreDumpDirectory" to some directory with write
permissions for the appropriate User in the httpd.conf and also setting the
"DAEMON_COREFILE_LIMIT" environment variable to "unlimited" in the
/etc/sysconfig/httpd file.  I assume it's not too different with Apache-2.2
and RHEL5.  If you can get Apache to dump core, you can backtrace with gdb.

I'm sorry I cannot be of more assistance, but I hope this helps somewhat.

--phess


-----Original Message-----
From: rt-users-bounces at lists.bestpractical.com
[mailto:rt-users-bounces at lists.bestpractical.com] On Behalf Of Aleksey
Tsalolikhin
Sent: Friday, January 21, 2011 4:26 PM
To: rt-users
Subject: Re: [rt-users] apache httpd segfaults

On Fri, Jan 21, 2011 at 4:12 PM, Aleksey Tsalolikhin
<atsaloli.tech at gmail.com> wrote:
> My RT 3.8.8 has lost its prettiness.  It's text-based now, no colors
> or graphics, only text.  It does work, but it isn't pretty.   No
> buttons, only links.
>
> The only errors I can find are in the Apache httpd log, and it seems
> each time I click on a link within RT, I get a message like this in my
> httpd error log:
>
> [Fri Jan 21 16:05:06 2011] [notice] child pid 32708 exit signal
> Segmentation fault (11)
>
> I use postgres 8.4 on the backend.
>
> Would appreciate any suggestions for troubleshooting this.


This is on CentOS 5.5 and Apache httpd 2.2

strace of the apache httpd process shows the following:
open("/usr/lib/perl5/5.8.8/PerlIO.pm", O_RDONLY) = 60
ioctl(60, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff0aee6850) = -1 ENOTTY
(Inappropriate ioctl for device)
lseek(60, 0, SEEK_CUR)                  = 0
read(60, "package PerlIO;\n\nour $VERSION = "..., 4096) = 4096
lseek(60, 382, SEEK_SET)                = 382
lseek(60, 0, SEEK_CUR)                  = 382
close(60)                               = 0
stat("/opt/rt3/bin/../local/lib/PerlIO/scalar.pmc", 0x7fff0aee6c00) =
-1 ENOENT (No such file or directory)
stat("/opt/rt3/bin/../local/lib/PerlIO/scalar.pm", 0x7fff0aee6ac0) =
-1 ENOENT (No such file or directory)
stat("/opt/rt3/bin/../lib/PerlIO/scalar.pmc", 0x7fff0aee6c00) = -1
ENOENT (No such file or directory)
stat("/opt/rt3/bin/../lib/PerlIO/scalar.pm", 0x7fff0aee6ac0) = -1
ENOENT (No such file or directory)
stat("/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/PerlIO/scal
ar.pmc",
0x7fff0aee6c00) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/PerlIO/scal
ar.pm",
0x7fff0aee6ac0) = -1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/site_perl/5.8.8/PerlIO/scalar.pmc",
0x7fff0aee6c00) = -1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/site_perl/5.8.8/PerlIO/scalar.pm",
0x7fff0aee6ac0) = -1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/site_perl/PerlIO/scalar.pmc", 0x7fff0aee6c00) =
-1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/site_perl/PerlIO/scalar.pm", 0x7fff0aee6ac0) = -1
ENOENT (No such file or directory)
stat("/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/PerlIO/sc
alar.pmc",
0x7fff0aee6c00) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/PerlIO/sc
alar.pm",
0x7fff0aee6ac0) = -1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/vendor_perl/5.8.8/PerlIO/scalar.pmc",
0x7fff0aee6c00) = -1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/vendor_perl/5.8.8/PerlIO/scalar.pm",
0x7fff0aee6ac0) = -1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/vendor_perl/PerlIO/scalar.pmc", 0x7fff0aee6c00) =
-1 ENOENT (No such file or directory)
stat("/usr/lib/perl5/vendor_perl/PerlIO/scalar.pm", 0x7fff0aee6ac0) =
-1 ENOENT (No such file or directory)
stat("/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/PerlIO/scalar.pmc",
0x7fff0aee6c00) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/PerlIO/scalar.pm",
{st_mode=S_IFREG|0644, st_size=979, ...}) = 0
open("/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/PerlIO/scalar.pm",
O_RDONLY) = 60
ioctl(60, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff0aee6850) = -1 ENOTTY
(Inappropriate ioctl for device)
lseek(60, 0, SEEK_CUR)                  = 0
read(60, "package PerlIO::scalar;\nour $VER"..., 4096) = 979
lseek(60, 108, SEEK_SET)                = 108
lseek(60, 0, SEEK_CUR)                  = 108
close(60)                               = 0
stat("/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/PerlIO/scalar/sc
alar.so",
{st_mode=S_IFREG|0755, st_size=11024, ...}) = 0
stat("/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/PerlIO/scalar/sc
alar.bs",
0x2b8fb08059a0) = -1 ENOENT (No such file or directory)
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
chdir("/etc/httpd")                     = 0
rt_sigaction(SIGSEGV, {SIG_DFL, [], SA_RESTORER|SA_INTERRUPT,
0x2b8f9056bb10}, {SIG_DFL, [], SA_RESTORER|SA_RESETHAND,
0x2b8f9056bb10}, 8) = 0
kill(8729, SIGSEGV)                     = 0
rt_sigreturn(0x2219)                    = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---


my httpd config for rt has not changed, I still have


    <Location /NoAuth/images>
        SetHandler default
    </Location>