[rt-users] AssetTracker crashes loading asset...raw horsepower solution?

[John]

9,359

On Tue, 11 Nov 2008, Todd Chapman wrote:
> Date: Tue, 11 Nov 2008 13:04:14 -0500
> From: Todd Chapman <todd at chaka.net>
> To: John <nimbius at sdf.lonestar.org>
> Cc: Curtis Bruneau <curtisb at vianet.ca>, rt-users at lists.bestpractical.com
> Subject: Re: [rt-users] AssetTracker crashes loading asset...raw horsepower
>     solution?
> 
> How many assets do you have? Nobody has ever seen this issue before....
>
> On Tue, Nov 11, 2008 at 12:57 PM, John <nimbius at sdf.lonestar.org> wrote:
>
>> On Fri, 7 Nov 2008, Curtis Bruneau wrote:
>>
>> I think i may have found a bit of a solution to the AT problem.  we're
>> running on 2 quad-core servers as frontends with 8gb memory each.  the
>> fault im seeing in AT is due to a timeout in apache waiting for its
>> backend processes to respond from huge queries...so:
>>
>>   <IfModule mod_fcgid.c>
>>     AddHandler fcgid-script .fcgi
>>
>> OutputBufferSize 1280000
>> IdleTimeout 600
>>   ProcessLifeTime 3600
>>   MaxProcessCount 8
>>   DefaultMinClassProcessCount 3
>>   DefaultMaxClassProcessCount 3
>>
>>   </IfModule>
>> IPCConnectTimeout 20
>> IPCCommTimeout 600
>>
>> has resolved the AT issue from what im seeing.  queries still take a long
>> time, but thats just because RT is pulling lots of other data from mysql
>> that im not certain is even pertanent to the asset/ticket at hand.  could
>> the speed of the query be increased by changing maxprocesscount to a
>> higher value?
>>
>> im running into issues now with the population of a user rights page that
>> includes 300-400 users...namely:
>>
>> RT::Group::Privileged Unimplemented in HTML::Mason::Commands.
>> (/opt/rt3/share/html/Elements/ShowUserConcise line 52)
>>
>> any ideas?
>>
>>
>>
>>> Date: Fri, 07 Nov 2008 12:52:04 -0500
>>> From: Curtis Bruneau <curtisb at vianet.ca>
>>> To: John <nimbius at sdf.lonestar.org>, rt-users at lists.bestpractical.com
>>> Subject: Re: [rt-users] AssetTracker crashes loading asset
>>>
>>> The queries would execute fine, the problem at least in my case is it
>> tries
>>> to load the whole result set into memory causing oom-killer to go on
>> rampage.
>>> Apache will peak out, to sort of help this situation I added a mem limit
>> to
>>> the fastcgi handler which will basically die before things get really
>> ugly.
>>> I'm able to see this query in the log, it has no limit clause so it
>> really is
>>> getting everything. While doing a form of debug/trace on the
>> apache/handler
>>> I'm also able to see it trying to load the full row times search results
>>> (table.*) into memory. This is all on a ticket display from a large
>> search
>>> result. So unless you are loading it in memory as opposed to say a shell
>>> client output you won't run into memory problems.
>>>
>>> Best of luck with your situation, I have not been able to get any
>>> confirmation until now with you. I've produced somewhat detailed bug
>> report
>>> but it appears to be somewhat isolated.. people with low tickets will
>> never
>>> run into this problem.
>>>
>>> Curtis
>>>
>>> John wrote:
>>>> curtis:
>>>> ive played around re-executing the mysql queries related to the RT
>> crash,
>>>> and to no avail.  they execute just fine.
>>>>
>>>> could this perhaps be a syslog issue like in RT?  is there a seperate
>>>> control for syslogging in AT?
>>>>
>>>>
>>>> On Fri, 7 Nov 2008, Curtis Bruneau wrote:
>>>>
>>>>> Date: Fri, 07 Nov 2008 10:43:48 -0500
>>>>> From: Curtis Bruneau <curtisb at vianet.ca>
>>>>> To: John <nimbius at sdf.lonestar.org>, rt-users at lists.bestpractical.com
>>>>> Subject: Re: [rt-users] AssetTracker crashes loading asset
>>>>>
>>>>> Sounds exactly like the issue I have, I think something is trying to
>> get
>>>>> all those records, I tried to trace it but with no luck, I think it may
>> be
>>>>> related to the back/next links on the ticket display page. It's
>> checking
>>>>> each record for something, This is ok with small results but crashes
>> with
>>>>> large sets. I really wish I could figure this one out, I get the
>>>>> occasional 500 and users are made aware to keep search results smaller.
>> In
>>>>> my testing this affected 3.4.x 3.6.x and 3.8.x
>>>>>
>>>>> John wrote:
>>>>>> well, just as i thought the RT slowness issue had been resolved,
>>>>>> assettracker
>>>>>> is now dying when loading a ticket out of a large list.
>>>>>>
>>>>>> example:  clicking "all assets" enumerates a 9000 row list fine,
>>>>>> but clicking on any element in the list will crunch for a while
>>>>>> then die with a 500 error.
>>>>>>
>>>>>> smaller lists with say 1000-3000 rows are sluggish when loading
>>>>>> an asset from the list, but succeed.
>>>>>>
>>>>>> nimbius at sdf.lonestar.org
>>>>>> SDF Public Access UNIX System - http://sdf.lonestar.org
>>>>>> _______________________________________________
>>>>>> http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users
>>>>>>
>>>>>> Community help: http://wiki.bestpractical.com
>>>>>> Commercial support: sales at bestpractical.com
>>>>>>
>>>>>>
>>>>>> Discover RT's hidden secrets with RT Essentials from O'Reilly Media.
>> Buy
>>>>>> a copy at http://rtbook.bestpractical.com
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>> nimbius at sdf.lonestar.org
>>>> SDF Public Access UNIX System - http://sdf.lonestar.org
>>>
>>
>> nimbius at sdf.lonestar.org
>> SDF Public Access UNIX System - http://sdf.lonestar.org
>> _______________________________________________
>> http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users
>>
>> Community help: http://wiki.bestpractical.com
>> Commercial support: sales at bestpractical.com
>>
>>
>> Discover RT's hidden secrets with RT Essentials from O'Reilly Media.
>> Buy a copy at http://rtbook.bestpractical.com
>>
>

nimbius at sdf.lonestar.org
SDF Public Access UNIX System - http://sdf.lonestar.org