[rt-users] Overnight segfault since RT 3.8.1 upgrade

[James Chamberlain]

On Mon, 15 Sep 2008, Jesse Vincent wrote:
>
> On Sep 15, 2008, at 8:55 AM, James Chamberlain wrote:
>
>> It does, but the thing that gets me is that this has only happened since I 
>> upgraded to 3.8.1.  If I switch back to 3.8.0 - granted, not something I 
>> should do since there have been database changes - I don't have the 
>> overnight segfault problem anymore.
>
> The database changes shouldn't be "dangerous", though that's quite 
> interesting. By "switch back" do you mean restoring the server to an older 
> version or just switching the RT directory?
>
> Can you catch a stacktrace from the segfault's core dump? Can you make it 
> dump core without waiting for 10 hours of silence from you users?

It definitely looks like it's MySQL-related now.  I set CoreDumpDirectory 
in my Apache configs and was greeted with a few cores this morning.  Taking 
a look at one of them (and skipping the "Reading" and "Loaded" symbols 
statements), I saw:

Core was generated by `/usr/sbin/httpd'.
Program terminated with signal 11, Segmentation fault.
#0  0x00002aaab160d81e in mysql_ping ()
    from /usr/lib64/mysql/libmysqlclient.so.15
(gdb) thread apply all bt full

Thread 1 (process 1029):
#0  0x00002aaab160d81e in mysql_ping ()
    from /usr/lib64/mysql/libmysqlclient.so.15
No symbol table info available.
#1  0x00002afc41cb1fde in XS_DBD__mysql__db_ping (
     my_perl=<value optimized out>, cv=<value optimized out>) at 
mysql.xs:554
 	dbh = (SV *) 0x2afc4667fe40
 	RETVAL = <value optimized out>
 	sp = <value optimized out>
 	ax = <value optimized out>
#2  0x00002afc39e94621 in XS_DBI_dispatch (my_perl=<value optimized out>,
     cv=0x2afc472c0be0) at DBI.xs:3287
 	markix = 0
 	xscv = (CV *) 0xe
 	sp = (SV **) 0x1
 	ax = 1
 	items = 1
 	perinterp_sv = <value optimized out>
 	PERINTERP = (PERINTERP_t *) 0x2afc4858cf00
 	h = (SV *) 0x2afc4667fe40
 	st1 = (SV *) 0x2afc472c0be0
 	st2 = (SV *) 0x2afc45e29820
 	err_sv = <value optimized out>
 	tmp_svp = <value optimized out>
 	hook_svp = (SV **) 0x2afc461d2440
 	mg = <value optimized out>
 	gimme = 0
 	trace_flags = 0
 	trace_level = 0
 	is_DESTROY = 0
 	is_unrelated_to_Statement = 1024
 	keep_error = 1
 	ErrCount = 0
 	i = <value optimized out>
 	outitems = 1
 	call_depth = 1
 	is_nested_call = 0
 	profile_t1 = 0
 	meth_name = 0x2afc472c9bd0 "ping"
 	ima = (const dbi_ima_t *) 0x2afc472cad10
 	ima_flags = <value optimized out>
 	imp_xxh = (imp_xxh_t *) 0x2afc339350c0
 	imp_msv = (SV *) 0x2afc338a40a0
 	qsv = (SV *) 0x2afc49eb38c0
#3  0x00002aaab2a289f6 in Perl_pp_entersub ()
    from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
No symbol table info available.
#4  0x00002aaab2a2229e in Perl_runops_standard ()
    from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
No symbol table info available.
#5  0x00002aaab29cf5f0 in Perl_call_sv ()
    from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
No symbol table info available.
#6  0x00002aaab2777b97 in modperl_callback ()
    from /etc/httpd/modules/mod_perl.so
No symbol table info available.
#7  0x00002aaab27782af in modperl_callback_run_handlers ()
    from /etc/httpd/modules/mod_perl.so
No symbol table info available.
#8  0x00002aaab27787ef in modperl_callback_per_dir ()
    from /etc/httpd/modules/mod_perl.so
No symbol table info available.
#9  0x00002aaab27728f0 in modperl_response_init ()
    from /etc/httpd/modules/mod_perl.so
No symbol table info available.
#10 0x00002aaab2772ab3 in modperl_response_handler_cgi ()
    from /etc/httpd/modules/mod_perl.so
No symbol table info available.
#11 0x00002afc3381f7ea in ap_run_handler () from /usr/sbin/httpd
No symbol table info available.
#12 0x00002afc33822c72 in ap_invoke_handler () from /usr/sbin/httpd
No symbol table info available.
#13 0x00002afc3382d5e8 in ap_process_request () from /usr/sbin/httpd
No symbol table info available.
#14 0x00002afc3382a870 in ap_register_input_filter () from /usr/sbin/httpd
No symbol table info available.
#15 0x00002afc33826a52 in ap_run_process_connection () from /usr/sbin/httpd
No symbol table info available.
#16 0x00002afc3383120b in ap_graceful_stop_signalled () from 
/usr/sbin/httpd
No symbol table info available.
#17 0x00002afc3383149a in ap_graceful_stop_signalled () from 
/usr/sbin/httpd
No symbol table info available.
#18 0x00002afc33831550 in ap_graceful_stop_signalled () from 
/usr/sbin/httpd
No symbol table info available.
#19 0x00002afc33832246 in ap_mpm_run () from /usr/sbin/httpd
No symbol table info available.
#20 0x00002afc3380ce04 in main () from /usr/sbin/httpd
No symbol table info available.
(gdb)

I presume this means I should try a different version of DBD::mysql.  I see 
that I installed DBD::mysql 4.008 as part of the upgrade to RT 3.8.1.  Does 
this sound right?  Anyone have any other thoughts?

Thanks,

James