[rt-users] AssetTracker crashes loading asset...raw horsepower solution?

Tue Nov 11 13:04:14 EST 2008

How many assets do you have? Nobody has ever seen this issue before....

On Tue, Nov 11, 2008 at 12:57 PM, John <nimbius at sdf.lonestar.org> wrote:

> On Fri, 7 Nov 2008, Curtis Bruneau wrote:
>
> I think i may have found a bit of a solution to the AT problem.  we're
> running on 2 quad-core servers as frontends with 8gb memory each.  the
> fault im seeing in AT is due to a timeout in apache waiting for its
> backend processes to respond from huge queries...so:
>
>   <IfModule mod_fcgid.c>
>     AddHandler fcgid-script .fcgi
>
> OutputBufferSize 1280000
> IdleTimeout 600
>   ProcessLifeTime 3600
>   MaxProcessCount 8
>   DefaultMinClassProcessCount 3
>   DefaultMaxClassProcessCount 3
>
>   </IfModule>
> IPCConnectTimeout 20
> IPCCommTimeout 600
>
> has resolved the AT issue from what im seeing.  queries still take a long
> time, but thats just because RT is pulling lots of other data from mysql
> that im not certain is even pertanent to the asset/ticket at hand.  could
> the speed of the query be increased by changing maxprocesscount to a
> higher value?
>
> im running into issues now with the population of a user rights page that
> includes 300-400 users...namely:
>
> RT::Group::Privileged Unimplemented in HTML::Mason::Commands.
> (/opt/rt3/share/html/Elements/ShowUserConcise line 52)
>
> any ideas?
>
>
>
> > Date: Fri, 07 Nov 2008 12:52:04 -0500
> > From: Curtis Bruneau <curtisb at vianet.ca>
> > To: John <nimbius at sdf.lonestar.org>, rt-users at lists.bestpractical.com
> > Subject: Re: [rt-users] AssetTracker crashes loading asset
> >
> > The queries would execute fine, the problem at least in my case is it
> tries
> > to load the whole result set into memory causing oom-killer to go on
> rampage.
> > Apache will peak out, to sort of help this situation I added a mem limit
> to
> > the fastcgi handler which will basically die before things get really
> ugly.
> > I'm able to see this query in the log, it has no limit clause so it
> really is
> > getting everything. While doing a form of debug/trace on the
> apache/handler
> > I'm also able to see it trying to load the full row times search results
> > (table.*) into memory. This is all on a ticket display from a large
> search
> > result. So unless you are loading it in memory as opposed to say a shell
> > client output you won't run into memory problems.
> >
> > Best of luck with your situation, I have not been able to get any
> > confirmation until now with you. I've produced somewhat detailed bug
> report
> > but it appears to be somewhat isolated.. people with low tickets will
> never
> > run into this problem.
> >
> > Curtis
> >
> > John wrote:
> >> curtis:
> >> ive played around re-executing the mysql queries related to the RT
> crash,
> >> and to no avail.  they execute just fine.
> >>
> >> could this perhaps be a syslog issue like in RT?  is there a seperate
> >> control for syslogging in AT?
> >>
> >>
> >> On Fri, 7 Nov 2008, Curtis Bruneau wrote:
> >>
> >>> Date: Fri, 07 Nov 2008 10:43:48 -0500
> >>> From: Curtis Bruneau <curtisb at vianet.ca>
> >>> To: John <nimbius at sdf.lonestar.org>, rt-users at lists.bestpractical.com
> >>> Subject: Re: [rt-users] AssetTracker crashes loading asset
> >>>
> >>> Sounds exactly like the issue I have, I think something is trying to
> get
> >>> all those records, I tried to trace it but with no luck, I think it may
> be
> >>> related to the back/next links on the ticket display page. It's
> checking
> >>> each record for something, This is ok with small results but crashes
> with
> >>> large sets. I really wish I could figure this one out, I get the
> >>> occasional 500 and users are made aware to keep search results smaller.
> In
> >>> my testing this affected 3.4.x 3.6.x and 3.8.x
> >>>
> >>> John wrote:
> >>>> well, just as i thought the RT slowness issue had been resolved,
> >>>> assettracker
> >>>> is now dying when loading a ticket out of a large list.
> >>>>
> >>>> example:  clicking "all assets" enumerates a 9000 row list fine,
> >>>> but clicking on any element in the list will crunch for a while
> >>>> then die with a 500 error.
> >>>>
> >>>> smaller lists with say 1000-3000 rows are sluggish when loading
> >>>> an asset from the list, but succeed.
> >>>>
> >>>> nimbius at sdf.lonestar.org
> >>>> SDF Public Access UNIX System - http://sdf.lonestar.org
> >>>> _______________________________________________
> >>>> http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users
> >>>>
> >>>> Community help: http://wiki.bestpractical.com
> >>>> Commercial support: sales at bestpractical.com
> >>>>
> >>>>
> >>>> Discover RT's hidden secrets with RT Essentials from O'Reilly Media.
> Buy
> >>>> a copy at http://rtbook.bestpractical.com
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >> nimbius at sdf.lonestar.org
> >> SDF Public Access UNIX System - http://sdf.lonestar.org
> >
>
> nimbius at sdf.lonestar.org
> SDF Public Access UNIX System - http://sdf.lonestar.org
> _______________________________________________
> http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users
>
> Community help: http://wiki.bestpractical.com
> Commercial support: sales at bestpractical.com
>
>
> Discover RT's hidden secrets with RT Essentials from O'Reilly Media.
> Buy a copy at http://rtbook.bestpractical.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bestpractical.com/pipermail/rt-users/attachments/20081111/9c0763bc/attachment.htm>