[rt-users] rt: Server error: Internal Server Error (500) for rt ls -i "LastUpdated >= '$(date -d "1 month ago")'"

[Trent W. Buck]

Something is seriously wrong with my RT deployment.  I first noticed the
problem with Ubuntu Hardy 8.04's RT 3.6 packages; yesterday I upgraded
to 3.8.1rc5 without noticeable improvement.  I also tried upgrading the
hardware to a 2.4GHz Celeron with 2Gb of RAM; this didn't seem to help.

I'm using the postgres backend and there are currently less than 200
tickets in my RT instance (this was a pilot deployment).

This (draft) bash script to rip timesheet data out of RT falls down and
dies:

    FIELDS=Ticket,Created,Creator,TimeTaken
    rt ls -i "LastUpdated >= '$(date -d "1 month ago")'" |
    sed 's|$|/history/type/Correspond|' |
    rt show -l -f "$FIELDS" - |
    egrep -v '^(#|$|--|RT/.* 200 Ok)' | # Remove lines that aren't data
    cut -d: -f 2- |                     # remove field names
    tr -d , |                           # kludge; comma is the delimiter
    paste -sd "${FIELDS//[!,]},\n"

It doesn't seem to always fail at the same point; right now it's failing
before it even gets to "rt show":

    $ rt ls -i "LastUpdated >= '$(date -I -d "1 week ago")'"
    ++ date -I -d '1 week ago'
    + rt ls -i 'LastUpdated >= '\''2008-08-14'\'''
    rt: Server error: Internal Server Error (500)

I strongly suspect this is because apache's perl interface (mod_perl2?)
is severing the connection because it took too long.

Possibly related, there are tens of thousands of lines in syslog of the
form:

    Aug 19 16:08:54 oyster RT: We found a merged ticket.3/66
    Aug 19 16:09:10 oyster last message repeated 376 times
    Aug 19 16:09:11 oyster RT: We found a merged ticket.4/65
    Aug 19 16:09:38 oyster last message repeated 686 times

These syslog messages seem to be appear when doing "rt show
ticket/$n/history", but not "rt ls -i $query".

The offending process seems to be CPU-bound for some time after rt(1)
fails, for example:

    PID   USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
    29445 www-data  20   0 63304  56m 4084 R 96.6  2.8  29:27.53
    /usr/bin/perl /usr/share/request-tracker3.8/libexec/mason_handler.fcgi