[rt-users] mod_fastcgi and mod_fcgid segfault's on certain tickets
Tomas Olaj
tomas.olaj at usit.uio.no
Fri Mar 23 05:53:26 EDT 2007
Hi,
We're running 3.6.1 in production (the same applies to 3.6.3 in test), and Pg
version 8.1.8, and FastCGI (mod_fcgi-2.4.2), and Perl 5.8.7.
Sometimes we experience tickets that only shows ticket metadata, and not the
whole ticket including history. Everything looks 'ok' in the database:
rttest=# SELECT a.id,a.type,b.name,a.created from transactions a left
join users b on a.creator = b.id where a.objectid = '71963' order by
a.created;
id | type | name | created
---------+--------------------+-----------+---------------------
284098 | Create | user | 2005-09-27 09:05:15
1391471 | Create | user | 2007-01-10 13:33:44
1391473 | EmailRecord | RT_System | 2007-01-10 13:33:45
1391472 | EmailRecord | RT_System | 2007-01-10 13:33:45
1391487 | Take | user | 2007-01-10 13:36:27
1391489 | Set | user | 2007-01-10 13:36:42
1391603 | EmailRecord | RT_System | 2007-01-10 13:49:25
1391602 | Status | RT_System | 2007-01-10 13:49:25
1391601 | Correspond | user | 2007-01-10 13:49:25
1392035 | Correspond | user | 2007-01-10 15:00:02
1392036 | EmailRecord | RT_System | 2007-01-10 15:00:03
1396227 | Comment | user | 2007-01-11 13:18:34
1396228 | CommentEmailRecord | RT_System | 2007-01-11 13:18:35
1396300 | Correspond | user | 2007-01-11 13:29:39
1396301 | EmailRecord | RT_System | 2007-01-11 13:29:40
1396329 | EmailRecord | RT_System | 2007-01-11 13:37:11
1396328 | Give | user | 2007-01-11 13:37:11
1399462 | Correspond | user | 2007-01-12 09:07:58
1399463 | EmailRecord | RT_System | 2007-01-12 09:07:59
(19 rows)
This is a serious problem.
A short excerpt from thestrace of the mason handler process (only
one running for the test) when a certain problem ticket try to load in
a web-browser. When the mason handler process segfault's, a new one is
created.
-->
time(NULL) = 1174556708
time(NULL) = 1174556708
stat64("/site/rt-3.6.3/local/lib/Text/Quoted.pmc", 0xbfffa550) = -1 ENOENT
(No such file or directory)
open("/site/rt-3.6.3/local/lib/Text/Quoted.pm", O_RDONLY|O_LARGEFILE) = -1
ENOENT (No such file or directory)
stat64("/site/rt-3.6.3/lib/Text/Quoted.pmc", 0xbfffa550) = -1 ENOENT (No
such file or directory)
open("/site/rt-3.6.3/lib/Text/Quoted.pm", O_RDONLY|O_LARGEFILE) = -1
ENOENT (No such file or directory)
stat64("/site/perl-5.8.7/lib/5.8.7/i686-linux/Text/Quoted.pmc",
0xbfffa550) = -1
ENOENT (No such file or directory)
open("/site/perl-5.8.7/lib/5.8.7/i686-linux/Text/Quoted.pm",
O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
stat64("/site/perl-5.8.7/lib/5.8.7/Text/Quoted.pmc", 0xbfffa550) = -1
ENOENT (No
such file or directory)
open("/site/perl-5.8.7/lib/5.8.7/Text/Quoted.pm", O_RDONLY|O_LARGEFILE) =
-1 ENOENT (No such file or directory)
stat64("/site/perl-5.8.7/lib/site_perl/5.8.7/i686-linux/Text/Quoted.pmc",
0xbfffa550) = -1 ENOENT (No such file or directory)
open("/site/perl-5.8.7/lib/site_perl/5.8.7/i686-linux/Text/Quoted.pm",
O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
stat64("/site/perl-5.8.7/lib/site_perl/5.8.7/Text/Quoted.pmc", 0xbfffa550)
= -1 ENOENT (No such file or directory)
open("/site/perl-5.8.7/lib/site_perl/5.8.7/Text/Quoted.pm",
O_RDONLY|O_LARGEFILE) = 9
ioctl(9, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfffa358) = -1 ENOTTY
(Inappropriate ioctl for device)
_llseek(9, 0, [0], SEEK_CUR) = 0
read(9, "package Text::Quoted;\nour $VERSI"..., 4096) = 4096
stat64("/site/rt-3.6.3/local/lib/Text/Autoformat.pmc", 0xbfffa0a0) = -1
ENOENT (No such file or directory)
open("/site/rt-3.6.3/local/lib/Text/Autoformat.pm", O_RDONLY|O_LARGEFILE)
= -1 ENOENT (No such file or directory)
stat64("/site/rt-3.6.3/lib/Text/Autoformat.pmc", 0xbfffa0a0) = -1 ENOENT
(No such file or directory)
open("/site/rt-3.6.3/lib/Text/Autoformat.pm", O_RDONLY|O_LARGEFILE) = -1
ENOENT (No such file or directory)
stat64("/site/perl-5.8.7/lib/5.8.7/i686-linux/Text/Autoformat.pmc",
0xbfffa0a0) = -1 ENOENT (No such file or directory)
open("/site/perl-5.8.7/lib/5.8.7/i686-linux/Text/Autoformat.pm",
O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
stat64("/site/perl-5.8.7/lib/5.8.7/Text/Autoformat.pmc", 0xbfffa0a0) = -1
ENOENT
(No such file or directory)
open("/site/perl-5.8.7/lib/5.8.7/Text/Autoformat.pm",
O_RDONLY|O_LARGEFILE) = -1
ENOENT (No such file or directory)
stat64("/site/perl-5.8.7/lib/site_perl/5.8.7/i686-linux/Text/Autoformat.pmc",
0xbfffa0a0) = -1 ENOENT (No such file or directory)
open("/site/perl-5.8.7/lib/site_perl/5.8.7/i686-linux/Text/Autoformat.pm",
O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
stat64("/site/perl-5.8.7/lib/site_perl/5.8.7/Text/Autoformat.pmc",
0xbfffa0a0) =
-1 ENOENT (No such file or directory)
open("/site/perl-5.8.7/lib/site_perl/5.8.7/Text/Autoformat.pm",
O_RDONLY|O_LARGEFILE) = 11
ioctl(11, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfff9ea8) = -1 ENOTTY
(Inappropriate i
octl for device)
_llseek(11, 0, [0], SEEK_CUR) = 0
read(11, "package Text::Autoformat;\n\nuse s"..., 4096) = 4096
stat64("/site/rt-3.6.3/local/lib/Text/Reform.pmc", 0xbfff9bf0) = -1 ENOENT
(No such file or directory)
open("/site/rt-3.6.3/local/lib/Text/Reform.pm", O_RDONLY|O_LARGEFILE) = -1
ENOENT (No such file or directory)
stat64("/site/rt-3.6.3/lib/Text/Reform.pmc", 0xbfff9bf0) = -1 ENOENT (No
such file or directory)
open("/site/rt-3.6.3/lib/Text/Reform.pm", O_RDONLY|O_LARGEFILE) = -1
ENOENT (No such file or directory)
stat64("/site/perl-5.8.7/lib/5.8.7/i686-linux/Text/Reform.pmc",
0xbfff9bf0) = -1
ENOENT (No such file or directory)
open("/site/perl-5.8.7/lib/5.8.7/i686-linux/Text/Reform.pm",
O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
stat64("/site/perl-5.8.7/lib/5.8.7/Text/Reform.pmc", 0xbfff9bf0) = -1
ENOENT (No
such file or directory)
open("/site/perl-5.8.7/lib/5.8.7/Text/Reform.pm", O_RDONLY|O_LARGEFILE) =
-1 ENOENT (No such file or directory)
stat64("/site/perl-5.8.7/lib/site_perl/5.8.7/i686-linux/Text/Reform.pmc",
0xbfff9bf0) = -1 ENOENT (No such file or directory)
open("/site/perl-5.8.7/lib/site_perl/5.8.7/i686-linux/Text/Reform.pm",
O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
stat64("/site/perl-5.8.7/lib/site_perl/5.8.7/Text/Reform.pmc", 0xbfff9bf0)
= -1 ENOENT (No such file or directory)
open("/site/perl-5.8.7/lib/site_perl/5.8.7/Text/Reform.pm",
O_RDONLY|O_LARGEFILE) = 12
ioctl(12, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfff99f8) = -1 ENOTTY
(Inappropriate ioctl for device)
_llseek(12, 0, [0], SEEK_CUR) = 0
read(12, "package Text::Reform;\n\nuse stric"..., 4096) = 4096
brk(0) = 0xab0d000
brk(0xab2e000) = 0xab2e000
read(12, "ess $config->{numeric} =~ m/\\bAl"..., 4096) = 4096
read(12, "$config->{footer}, $config->{pag"..., 4096) = 4096
brk(0) = 0xab2e000
brk(0xab4f000) = 0xab4f000
read(12, "]}||\"\"\n\t\t\t\t: ${$_[$startidx++]}|"..., 4096) = 4096
read(12, "oter.=\"\\n\"\n\t\t\t\tif $lastfooter &&"..., 4096) = 4096
_llseek(12, 19355, [19355], SEEK_SET) = 0
_llseek(12, 0, [19355], SEEK_CUR) = 0
close(12) = 0
read(11, ";\n\n\t# SPECIAL IGNORANCE...\n\tif ("..., 4096) = 4096
stat64("/site/rt-3.6.3/local/lib/Text/Tabs.pmc", 0xbfff9bf0) = -1 ENOENT
(No such file or directory)
open("/site/rt-3.6.3/local/lib/Text/Tabs.pm", O_RDONLY|O_LARGEFILE) = -1
ENOENT (No such file or directory)
stat64("/site/rt-3.6.3/lib/Text/Tabs.pmc", 0xbfff9bf0) = -1 ENOENT (No
such file
or directory)
open("/site/rt-3.6.3/lib/Text/Tabs.pm", O_RDONLY|O_LARGEFILE) = -1 ENOENT
(No such file or directory)
stat64("/site/perl-5.8.7/lib/5.8.7/i686-linux/Text/Tabs.pmc", 0xbfff9bf0)
= -1 ENOENT (No such file or directory)
open("/site/perl-5.8.7/lib/5.8.7/i686-linux/Text/Tabs.pm",
O_RDONLY|O_LARGEFILE)
= -1 ENOENT (No such file or directory)
stat64("/site/perl-5.8.7/lib/5.8.7/Text/Tabs.pmc", 0xbfff9bf0) = -1 ENOENT
(No such file or directory)
or directory)
open("/site/rt-3.6.3/lib/Text/Tabs.pm", O_RDONLY|O_LARGEFILE) = -1 ENOENT
(No such file or directory)
stat64("/site/perl-5.8.7/lib/5.8.7/i686-linux/Text/Tabs.pmc", 0xbfff9bf0)
= -1 ENOENT (No such file or directory)
open("/site/perl-5.8.7/lib/5.8.7/i686-linux/Text/Tabs.pm",
O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
stat64("/site/perl-5.8.7/lib/5.8.7/Text/Tabs.pmc", 0xbfff9bf0) = -1 ENOENT
(No such file or directory)
open("/site/perl-5.8.7/lib/5.8.7/Text/Tabs.pm", O_RDONLY|O_LARGEFILE) = 12
ioctl(12, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfff99f8) = -1 ENOTTY
(Inappropriate ioctl for device)
_llseek(12, 0, [0], SEEK_CUR) = 0
read(12, "\npackage Text::Tabs;\n\nrequire Ex"..., 4096) = 2291
brk(0) = 0xab4f000
brk(0xab70000) = 0xab70000
_llseek(12, 1098, [1098], SEEK_SET) = 0
_llseek(12, 0, [1098], SEEK_CUR) = 0
close(12) = 0
read(11, "\t}\n\t\t\tif ($args{case} =~ /highli"..., 4096) = 4096
read(11, "$tfield\";\n\t\t $newtext = (!$pa"..., 4096) = 4096
read(11, "ngth $2;\n\tmy $aindent = length $"..., 4096) = 4096
brk(0) = 0xab70000
brk(0xab91000) = 0xab91000
read(11, "ev) {\t# going down level(s)\n\t\tfo"..., 4096) = 4096
_llseek(11, 22231, [22231], SEEK_SET) = 0
_llseek(11, 0, [22231], SEEK_CUR) = 0
close(11) = 0
read(9, "low\" where we are right\n "..., 4096) = 3601
read(9, "", 4096) = 0
close(9) = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
<--
This was testet using mod_fastcgi. We also tested mod_fcgi (latest
snapshot) to see if we could reproduce the problem.
It behaved different. Instead of only showing metadata as the mod_fastcgi
did, we just got an "Internal Server Error" in web browser testing the
same
ticket.
The httpd.conf configuration for fcgid:
# This is for mod_fastcgi
# Tell FastCGI to put its temporary files somewhere sane.
# FastCgiIpcDir /tmp
# FastCgiServer /site/rttest/bin/mason_handler.fcgi -idle-timeout 300 -processes 1
# This is for mod_fcgid
SocketPath /tmp/fcgidsock
SharememPath /tmp/fcgidshm
DefaultInitEnv RAILS_ENV production
IdleTimeout 300
MaxProcessCount 1
FCGIWrapper "/site/rttest/bin/mason_handler.fcgi" .fcgi
>From Apache error.log:
-->
[Fri Mar 23 10:29:16 2007] [warn] mod_fcgid: Read data error, fastcgi
server has close connection
[Fri Mar 23 10:29:16 2007] [warn] (32)Broken pipe: mod_fcgid:
ap_pass_brigade failed in handle_request function
[Fri Mar 23 10:30:44 2007] [warn] mod_fcgid: Read data error, fastcgi
server has close connection
[Fri Mar 23 10:30:44 2007] [warn] (32)Broken pipe: mod_fcgid:
ap_pass_brigade failed in handle_request function
[Fri Mar 23 10:32:43 2007] [warn] mod_fcgid: Read data error, fastcgi
server has close connection
[Fri Mar 23 10:32:43 2007] [warn] (32)Broken pipe: mod_fcgid:
ap_pass_brigade failed in handle_request function
[Fri Mar 23 10:41:00 2007] [warn] mod_fcgid: Read data error, fastcgi
server has close connection
[Fri Mar 23 10:41:00 2007] [warn] (32)Broken pipe: mod_fcgid:
ap_pass_brigade failed in handle_request function
<--
Anyone with similar experience or suggestions?
We have not tested the problem ticket crashing Fast CGI modules on
mod_perl yet. It seems that there may be some strange combination in
tichet that perhaps could crash Fast CGI??
All comments are appreciated a lot!
best regards and cheers,
Tomas
--
________________________________________________________________________
Tomas A. P. Olaj, email: tomas.olaj at usit.uio.no, web: folk.uio.no/tomaso
University of Oslo / USIT (Center for Information Technology Services)
System- and Application Management / Applications Management Group
More information about the rt-users
mailing list