[rt-users] RT suddenly no longer works (blank titleless page for everything, no errors in apache)

[Joel Feiner]

Hello,

We have had a successfully operating RT installing for many months now.  
This weekend, we upgraded perl and now RT no longer works at all.  The 
upgrade of perl was rolled back (it was just a minor update, so it 
shouldn't have caused a major problem, in any case).  The problem 
persists.  We have tried reinstalling RT, enabling debugging and not 
only does RT no longer work, but there are no errors in the Apache logs 
and the debugging log I set up in /opt/rt3/var/log remains empty.  None 
of the configs changed prior or during the outage, except to add in 
debugging options, which have had no visible effect.

Because our RT system is behind HTTPS, and also load-balanced using PEN, 
it is hard to test with, e.g. telnet or bin/rt.  Note, of course, that 
it worked successfully using this configuration before.

We are using RT 3.6.5 on CentOS 5 (x86_64) running inside two Xen 
virtual machines, themselves running on CentOS (x86_64).  Perl version 
is 5.8.8 (build 10), mod_perl is version 2.0.2 (build 6.3).  All CPAN 
dependencies are okay, verified with sbin/rt-test-dependencies.

Here are the relevant config files:
/opt/rt3/etc/RT_SiteConfig.pm:

Set( $rtname, 'XXXXX');
Set( $Organization, "XXXXXX");
Set( $OWnerEmail , 'XXXXXX');
Set( $WebBaseURL, "https://XXXXXXX");
Set( $Timezone, "US/Eastern");
Set( $WebExternalAuth, 1);
Set($DatabaseHost   , 'XXXXX');
Set($DatabaseRTHost , 'XXXXX');
Set($LogToFileName, "rt.log");
Set($LogToFile, "debug");
1;

/etc/httpd/conf/httpd.conf (relevant section)

<VirtualHost XXXXX:443>
        ServerName XXXXX
        DocumentRoot /opt/rt3/share/html
        AddDefaultCharset UTF-8

  PerlRequire /opt/rt3/bin/webmux.pl

  <Location />
    SetHandler perl-script
    PerlHandler RT::Mason
  </Location>

  SSLEngine on
  SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW

  SSLCertificateFile XXXXXX
  SSLCertificateKeyFile XXXXXX
  SSLCertificateChainFile XXXXXX

        ErrorLog /var/log/httpd/error_log
        TransferLog /var/log/httpd/access_log
RedirectMatch permanent (.*)/$ https://XXXXXX$1/index.html
</VirtualHost>

/etc/httpd/conf.d/perl.conf

# <comments removed>

LoadModule perl_module modules/mod_perl.so

# and a large number of commented out things [we haven't otherwise 
modified this file]

So, at this point, we need some advice on the direction to take, and 
what things to try.  I don't know enough about RT internals to properly 
debug this situation.  I've read some of the documentation, but again, 
since we didn't change any config before this broke, it shouldn't make a 
difference.

Thanks in advance.

- Joel