[rt-users] Testing 4.0.0rc5 - experiences upgrading from 3.6.4 on RHEL6

[Darren Nickerson]

On Feb 22, 2011, at 1:31 PM, Jesse Vincent wrote:
> 
> Darren,
> 
> Can you try out this new version of Plack:
> 
> http://pause.perl.org/incoming/Plack-0.9970.tar.gz
> 
> We believe it may resolve your issues.

Jesse,

Thanks for your help!

I installed that yesterday, but had to restart apache about 6 hours later because rt-mailgate was failing to insert tickets correctly. Well, actually the email was getting added to the ticket, but postfix thought it was failing:

Feb 22 21:47:34 rt4 postfix/local[20829]: E2AE4C0672: to=<support_ifax at rt4.corp.ifax.com>, relay=local, delay=943, delays=762/0.02/0/180, dsn=4.3.0, status=deferred (temporary failure. Command output: An Error Occurred =================  500 read timeout )
Feb 22 22:07:35 rt4 postfix/local[20920]: E2AE4C0672: to=<support_ifax at rt4.corp.ifax.com>, relay=local, delay=2143, delays=1963/0.02/0/180, dsn=4.3.0, status=deferred (temporary failure. Command output: An Error Occurred =================  500 read timeout )

That's just one 40-minute retry interval there, it kept injecting this transaction (and others) every 40 mins until I restarted apache. This seems like the same behavior I was seeing before installing this version of Plack.

Just to give you a bit more information about what I know here, I don't see this when RT is busy, ... it's usually in the evenings when it's been idle. And these read timeouts aren't limited to rt-mailgate, as I mentioned the user just sees a seemingly eternal hang when trying to access RT via the web as well. One time when this happened I kept hitting reload about 20 times on the browser, and noticed that each time I did so an apache thread went into a "W" (sending reply) status as reported by mod_status. Not getting any joy there, I tried 'apachectl graceful' to restart httpd, and as you can see below each thread just went into a "G" state, but stayed there until I stopped apache and restarted it.

I'm very interested in getting to the bottom of this and will help any way I can ... I'm just not entirely sure what to try next, or how to get any visibility into where the lockup is occurring.

Oh, and I'm still seeing these in the apache error log:

[Wed Feb 23 07:31:14 2011] [error] Your request path is '/NoAuth/Logout.html' and it doesn't match your Location(Match) '/opt/rt4/share/html/'. This should be due to the configuration error. See perldoc Plack::Handler::Apache2 for details.
[Wed Feb 23 07:45:16 2011] [error] Your request path is '/REST/1.0/NoAuth/mail-gateway' and it doesn't match your Location(Match) '/opt/rt4/share/html/'. This should be due to the configuration error. See perldoc Plack::Handler::Apache2 for details.
[Wed Feb 23 07:46:14 2011] [error] Your request path is '/robots.txt' and it doesn't match your Location(Match) '/opt/rt4/share/html/'. This should be due to the configuration error. See perldoc Plack::Handler::Apache2 for details.

-Darren


[root at rt4 ~]# apachectl status
                       Apache Server Status for localhost

   Server Version: Apache/2.2.15 (Unix) DAV/2 PHP/5.3.2 mod_ssl/2.2.15
   OpenSSL/1.0.0-fips mod_wsgi/3.2 Python/2.6.5 mod_perl/2.0.4 Perl/v5.10.1

   Server Built: Aug 14 2010 08:53:48

   --------------------------------------------------------------------------

   Current Time: Monday, 21-Feb-2011 11:15:41 EST

   Restart Time: Sunday, 20-Feb-2011 03:09:24 EST

   Parent Server Generation: 4

   Server uptime: 1 day 8 hours 6 minutes 17 seconds

   Total accesses: 887 - Total Traffic: 16.3 MB

   CPU Usage: u128.81 s53.18 cu0 cs0 - .157% CPU load

   .00767 requests/sec - 147 B/second - 18.8 kB/request

   34 requests currently being processed, 7 idle workers

 GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG_G__W____.......................
 ................................................................
 ................................................................
 ................................................................

   Scoreboard Key:
   "_" Waiting for Connection, "S" Starting up, "R" Reading Request,
   "W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
   "C" Closing connection, "L" Logging, "G" Gracefully finishing,
   "I" Idle cleanup of worker, "." Open slot with no current process
[root at rt4 ~]# uptime
 11:15:44 up 2 days, 22 min,  2 users,  load average: 1.18, 0.37, 0.13
[root at rt4 ~]# 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bestpractical.com/pipermail/rt-users/attachments/20110223/e551ce08/attachment.htm>