[rt-devel] Handle.pm too many clients

[ivan]

On Wed, Mar 21, 2001 at 07:57:08AM +0000, Feargal Reilly wrote:
>      
>      On Wed, Mar 21, 2001 at 07:28:28AM +0000, Feargal Reilly wrote:
>      > It's not so much apache keeping the connections I'd suspect, as mod_perl. 
>      
>      Sorry. I spoke imprecisely.

Apache::DBI, in fact, included in bin/webmux.pl.  I kinda think it
shouldn't be there, and the local admin or packager (for .rpm or .deb or
whatnot) should enable it with `PerlModule Apache::DBI' in their Apache
config.  I know it threw me for a loop when removing it from my httpd.conf
didn't work (I'm used to swapping out Apache::DBI for DBIx::Profile in my
Apache config)  Anyway, I have to give this some more thought.

I've actually been meaning to comment on other connection caching stuff
that I am a bit more sure of: 

DBIx::SearchBuilder calls the DBI method connect_cached in Handle.pm and
Handle/Oracle.pm.  I suggest this be changed to the regular DBI connect 
method.  The reasons for this are as follows:

- See the connect_cached documentation in the DBI manpage, which warns
that connect_cached is different than Apache::DBI and additionally should
be used with care.  Since DBIx::SearchBuilder is intended as general
purpose poop[1] it should default to use connect, not connect_cached.

- Since the only persistant context for RT is the web interface (the
mailgate and CLI start up a process and a new database connection), and
persistance in this context is better handled by Apache::DBI, RT should
have no need to ask DBIx::SearchBuilder for connect_cached.

>      > The problem is, each new client (read click) opens two new connections, so with 5 clients (clicks), I've 10 persistent connection processes running.
>      > Five hours later, they're still running. Now all I have to do is login again, view a couple more tickets, and I max out my clients, even though I'm the only user.
>      > Instead of reusing an existing connection, it's gone and done the whole construction all over again. When does it decide to open a new connection in the code?
>      
>      It gets one and only one per child. I don't know how apache
>      decides to hand your request to a specific child...

This has actually been a subject of much discussion.  It's called the
"thundering herd problem" - see
http://www.citi.umich.edu/projects/linux-scalability/reports/accept.html
for example. 

>      > For postgres, I am allowing 64 open connections, double the default, and way more than neccessary. 
>      > For apache, I've allowed 250 MaxClients, a figure which should never, ever be reached. 
>      
>      If you want to have up to 250 apache processes running, then you need to tune
>      your database to handle that.
>      
> There's no correlation between apache processes and database connections - not everything requires a db connection.

I think a better architecture is a "backend" mod_perl Apache server for
RT, with a much smaller MaxClients than your regular Apache.
http://perl.apache.org/guide/scenario.html has examples for a variety of
multiple-Apache configurations.

Personally, since I don't have the need to have RT show up in the regular
document tree or anything fancy like that, I just run an Apache process
for RT on a separate IP address (you could get the same functionality
using a separate port).

On the subject of tuning your database (since I've been bitten by this
problem): you want to set your database concurrent connections to: 
MaxClients from the Apache config, plus the maximum number of concurrent
local processes spawned by your MTA (i.e. maximum number of concurrent RT
mailgate processes), plus enough additional connections for command-line
processes and anything else that uses the same database. 

To control the maximum number of processes spawned by your MTA:

qmail: concurrencylocal (see the qmail-send manpage)
postfix: http://www.postfix.org/rate.html#process (you want to set the
         process limit for local delivery)
courier: http://www.courier-mta.org/modules.html (you want to set MAXDELS
         for the `local' module)
sendmail: no way to control this specifically.  you are forced to use
          sendmail's limit on the total number of
          processes (for any purpose): MaxDaemonProcesses in sendmail.cf

[1] "perl object oriented persistance" - see
    <http://sourceforge.net/mail/?group_id=4582>.  What were *you*
    thinking?

-- 
meow
_ivan