How many assets do you have? Nobody has ever seen this issue before....<br><br><div class="gmail_quote">On Tue, Nov 11, 2008 at 12:57 PM, John <span dir="ltr"><<a href="mailto:nimbius@sdf.lonestar.org">nimbius@sdf.lonestar.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">On Fri, 7 Nov 2008, Curtis Bruneau wrote:<br>
<br>
I think i may have found a bit of a solution to the AT problem. we're<br>
running on 2 quad-core servers as frontends with 8gb memory each. the<br>
fault im seeing in AT is due to a timeout in apache waiting for its<br>
backend processes to respond from huge queries...so:<br>
<br>
<IfModule mod_fcgid.c><br>
AddHandler fcgid-script .fcgi<br>
<br>
OutputBufferSize 1280000<br>
IdleTimeout 600<br>
ProcessLifeTime 3600<br>
MaxProcessCount 8<br>
DefaultMinClassProcessCount 3<br>
DefaultMaxClassProcessCount 3<br>
<br>
</IfModule><br>
IPCConnectTimeout 20<br>
IPCCommTimeout 600<br>
<br>
has resolved the AT issue from what im seeing. queries still take a long<br>
time, but thats just because RT is pulling lots of other data from mysql<br>
that im not certain is even pertanent to the asset/ticket at hand. could<br>
the speed of the query be increased by changing maxprocesscount to a<br>
higher value?<br>
<br>
im running into issues now with the population of a user rights page that<br>
includes 300-400 users...namely:<br>
<br>
RT::Group::Privileged Unimplemented in HTML::Mason::Commands.<br>
(/opt/rt3/share/html/Elements/ShowUserConcise line 52)<br>
<br>
any ideas?<br>
<br>
<br>
<br>
> Date: Fri, 07 Nov 2008 12:52:04 -0500<br>
> From: Curtis Bruneau <<a href="mailto:curtisb@vianet.ca">curtisb@vianet.ca</a>><br>
> To: John <<a href="mailto:nimbius@sdf.lonestar.org">nimbius@sdf.lonestar.org</a>>, <a href="mailto:rt-users@lists.bestpractical.com">rt-users@lists.bestpractical.com</a><br>
> Subject: Re: [rt-users] AssetTracker crashes loading asset<br>
><br>
> The queries would execute fine, the problem at least in my case is it tries<br>
> to load the whole result set into memory causing oom-killer to go on rampage.<br>
> Apache will peak out, to sort of help this situation I added a mem limit to<br>
> the fastcgi handler which will basically die before things get really ugly.<br>
> I'm able to see this query in the log, it has no limit clause so it really is<br>
> getting everything. While doing a form of debug/trace on the apache/handler<br>
> I'm also able to see it trying to load the full row times search results<br>
> (table.*) into memory. This is all on a ticket display from a large search<br>
> result. So unless you are loading it in memory as opposed to say a shell<br>
> client output you won't run into memory problems.<br>
><br>
> Best of luck with your situation, I have not been able to get any<br>
> confirmation until now with you. I've produced somewhat detailed bug report<br>
> but it appears to be somewhat isolated.. people with low tickets will never<br>
> run into this problem.<br>
><br>
> Curtis<br>
><br>
> John wrote:<br>
>> curtis:<br>
>> ive played around re-executing the mysql queries related to the RT crash,<br>
>> and to no avail. they execute just fine.<br>
>><br>
>> could this perhaps be a syslog issue like in RT? is there a seperate<br>
>> control for syslogging in AT?<br>
>><br>
>><br>
>> On Fri, 7 Nov 2008, Curtis Bruneau wrote:<br>
>><br>
>>> Date: Fri, 07 Nov 2008 10:43:48 -0500<br>
>>> From: Curtis Bruneau <<a href="mailto:curtisb@vianet.ca">curtisb@vianet.ca</a>><br>
>>> To: John <<a href="mailto:nimbius@sdf.lonestar.org">nimbius@sdf.lonestar.org</a>>, <a href="mailto:rt-users@lists.bestpractical.com">rt-users@lists.bestpractical.com</a><br>
>>> Subject: Re: [rt-users] AssetTracker crashes loading asset<br>
>>><br>
>>> Sounds exactly like the issue I have, I think something is trying to get<br>
>>> all those records, I tried to trace it but with no luck, I think it may be<br>
>>> related to the back/next links on the ticket display page. It's checking<br>
>>> each record for something, This is ok with small results but crashes with<br>
>>> large sets. I really wish I could figure this one out, I get the<br>
>>> occasional 500 and users are made aware to keep search results smaller. In<br>
>>> my testing this affected 3.4.x 3.6.x and 3.8.x<br>
>>><br>
>>> John wrote:<br>
>>>> well, just as i thought the RT slowness issue had been resolved,<br>
>>>> assettracker<br>
>>>> is now dying when loading a ticket out of a large list.<br>
>>>><br>
>>>> example: clicking "all assets" enumerates a 9000 row list fine,<br>
>>>> but clicking on any element in the list will crunch for a while<br>
>>>> then die with a 500 error.<br>
>>>><br>
>>>> smaller lists with say 1000-3000 rows are sluggish when loading<br>
>>>> an asset from the list, but succeed.<br>
>>>><br>
>>>> <a href="mailto:nimbius@sdf.lonestar.org">nimbius@sdf.lonestar.org</a><br>
>>>> SDF Public Access UNIX System - <a href="http://sdf.lonestar.org" target="_blank">http://sdf.lonestar.org</a><br>
>>>> _______________________________________________<br>
>>>> <a href="http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users" target="_blank">http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users</a><br>
>>>><br>
>>>> Community help: <a href="http://wiki.bestpractical.com" target="_blank">http://wiki.bestpractical.com</a><br>
>>>> Commercial support: <a href="mailto:sales@bestpractical.com">sales@bestpractical.com</a><br>
>>>><br>
>>>><br>
>>>> Discover RT's hidden secrets with RT Essentials from O'Reilly Media. Buy<br>
>>>> a copy at <a href="http://rtbook.bestpractical.com" target="_blank">http://rtbook.bestpractical.com</a><br>
>>>><br>
>>>><br>
>>>><br>
>>><br>
>>><br>
>><br>
>> <a href="mailto:nimbius@sdf.lonestar.org">nimbius@sdf.lonestar.org</a><br>
>> SDF Public Access UNIX System - <a href="http://sdf.lonestar.org" target="_blank">http://sdf.lonestar.org</a><br>
><br>
<br>
<a href="mailto:nimbius@sdf.lonestar.org">nimbius@sdf.lonestar.org</a><br>
SDF Public Access UNIX System - <a href="http://sdf.lonestar.org" target="_blank">http://sdf.lonestar.org</a><br>
_______________________________________________<br>
<a href="http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users" target="_blank">http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users</a><br>
<br>
Community help: <a href="http://wiki.bestpractical.com" target="_blank">http://wiki.bestpractical.com</a><br>
Commercial support: <a href="mailto:sales@bestpractical.com">sales@bestpractical.com</a><br>
<br>
<br>
Discover RT's hidden secrets with RT Essentials from O'Reilly Media.<br>
Buy a copy at <a href="http://rtbook.bestpractical.com" target="_blank">http://rtbook.bestpractical.com</a><br>
</blockquote></div><br>