[rt-users] "Out of memory" error in logs when restarting apache

[Greg]

Jaime Kikpole wrote:

>     Whenever I use mod_perl since my 2.0.15 to 3.2.2 upgrade (which involved
>a 3.0.x upgrade in the middle), I get errors like this in my
>/var/log/apache_error_log file and find that apache isn't running.
>  
>
Can you start Apache single-threaded, with ' -X'? Perhaps by adding a stanza
like this to the start script? I scribbled this in for Linux, not 
FreeBSD, but you
get the general idea.

  start-debug() {
          echo $"Starting $prog in DEBUG mode: "
          check13 || exit 1
          LANG=$HTTPD_LANG daemon $httpd -X $OPTIONS && echo -n "Exiting 
... "
  }

After getting a single-threaded Apache, I switch to another (much 
larger) window
and attach strace to it. On FreeBSD, you can use ktrace instead.

  (start Apache in debug mode)
  (get the PID of the httpd)
  ktrace -f /tmp/apache.kd -p $PID

Then you can try to reproduce the error you saw in your logs, using whatever
combination of page-loads or logins that it required.

Stop the tracing with ktrace -C after you see the bug.

Read the data with kdump. Find out what was happening while the 18MB
of data were being allocated (big sbrk()). Watch the file i/o, since 
that is how
the app communicates with the database. Is that memory being malloc'ed 
to hold
some large part of your database? Look for the last SQL query before the
problem.

Caveats: The problem you mention may require a multi-process Apache. Using
a single-process Apache to debug your problem could make the bug impossible
to reproduce.

Once again, use

  ktrace -C

to stop the tracing to that file so that your disk does not fill up with 
dump data. :)

That last part is pretty important.

Do let us know what you find.

Greg