[rt-users] RT 2.0.14 hangs on certain queries

[Michael Bilow]

We have a Debian "sarge" (testing) system that was working for some time,
using the Debian package "request-tracker" 2.0.14-2 that has been orphaned
to make way for RT3.  We did not upgrade to RT3 mainly because the system
has been working for us and we were concerned about the migration path for
the existing database.  Can I install RT3 and have it share access to the
same database used for RT2, or is there a one-way migration required?

We were primarily using the web interface with Apache 1.3.27, mod_perl
1.27, HTML::Mason 1.21, and perl 5.8.0.  This was a working system, but
since we are running on the Debian testing distribution, it is possible
that the package manager swapped something subtle that we didn't notice.

The only slightly unusual thing about our installation is that we access
the database over the network.  As explained, this has been working for
some time as well.  The database server machine is running the Debian
"woody" (stable) distribution, with package "postgresql" 7.2.1-2woody2.

Yesterday, some queries from the web interface started hanging.  For
example, if a user tries to log in with the wrong password, the web site
(correctly) returns an error immediately after consulting the database.  
However, if the user logs in with the correct password, the web site hangs
and the server shows the relevant instance of Apache pegged approaching
100% of CPU utilization.  In order to stop the runaway instance, we need
to kill and restart Apache.

The "rt-mailgate" tool continues to work, and users continue to be able to
submit tickets and get responses via e-mail.

Looking into this further, I discovered that the "rt" command line utility
hangs on some types of queries and not others.  For example, a relatively
complicated query with a bunch of constraints like this succeeds:

	./rt --limit-owner=mikebw --limit-status=new --limit-status=open
	   --limit-last-update=20031009- --summary

However, adding a constraint for requestor causes a hang until killed with
Ctrl-C.  In fact, even a constraint for only the requestor hangs:

	./rt '--limit-requestor=mike at bilow.com' --summary

My inference is that the web site hangs after logging in because one of
the regions that is displayed to a user on their RT home is the list of
tickets for which they are the requstor.

The fact that the "rt" command line tool hangs combined with the fact that
web site works in some cases indicates to me that the problem is somewhere
inside the Perl code and not in the Apache-specific stuff such as mod_perl
and Mason, but I suppose I could be wrong about this.  Regardless, since I
have a failure at the command line, I can use tools such as strace to
examime what is going on.

Using strace to compare the cases where the request does not hang as
opposed to where it does hang, both open the log files in the standard
place used by the Debian package, /var/log/request-tracker/.  Oddly, in
both cases, the log files are zero-length.  The last shared activity
between the non-hanging and hanging instances is some stuff about the
owner constraint on both queries:

 rt_sigaction(SIGPIPE, {SIG_IGN}, {SIG_DFL}, 8) = 0
 send(3, "QSELECT  * FROM Users WHERE lower(Name) = \'mikebw\'\0", 51,
0) = 51
 rt_sigaction(SIGPIPE, {SIG_DFL}, {SIG_IGN}, 8) = 0
 select(4, [3], [], [3], NULL)           = 1 (in [3])
 recv(3,
"Pblank\0T\0\"id\0\0\0\0\27\0\4\377\377\377\377name\0\0\0\4\23\377\377\
0\0\0|password\0\0\0\4\23\377\377\0\0\0,comments\0\0\0\0\31\377\377\377\377\377\
377signature\0\0\0\0\31\377\377\377\377\377\377emailaddress\0\0\0\4\23\377\377\0
\0\0|freeformc"..., 16384, 0) = 951
 rt_sigaction(SIGPIPE, {SIG_IGN}, {SIG_DFL}, 8) = 0
 send(3, "QSELECT  * FROM Users WHERE id = \'4\'\0", 37, 0) = 37
 rt_sigaction(SIGPIPE, {SIG_DFL}, {SIG_IGN}, 8) = 0
 select(4, [3], [], [3], NULL)           = 1 (in [3])
 recv(3,
"Pblank\0T\0\"id\0\0\0\0\27\0\4\377\377\377\377name\0\0\0\4\23\377\377\
0\0\0|password\0\0\0\4\23\377\377\0\0\0,comments\0\0\0\0\31\377\377\377\377\377\
377signature\0\0\0\0\31\377\377\377\377\377\377emailaddress\0\0\0\4\23\377\377\0
\0\0|freeformc"..., 16384, 0) = 951

Following this, both instances write the header to the stdout console:

 brk(0x87f3000)                          = 0x87f3000
 brk(0)                                  = 0x87f3000
 write(1, "  id  Stat    Queue                                   Subject
  Requestor \n", 81) = 81

But then the non-hanging version begins sending and receiving database
queries (the leading "-" on each line comes from "diff -u" used for this
comparison of the strace logs):

-rt_sigaction(SIGPIPE, {SIG_IGN}, {SIG_DFL}, 8) = 0
-send(3, "QSELECT main.* FROM Tickets main   WHERE ((main.EffectiveId =
main.id)) AND ((main.Owner = \'4\')) AND ((main.Status = \'new\')OR(ma"..., 151,
0) = 151
-rt_sigaction(SIGPIPE, {SIG_DFL}, {SIG_IGN}, 8) = 0
-select(4, [3], [], [3], NULL)           = 1 (in [3])
-recv(3,
"Pblank\0T\0\30id\0\0\0\0\27\0\4\377\377\377\377effectiveid\0\0\0\0\27\
0\4\377\377\377\377queue\0\0\0\0\27\0\4\377\377\377\377type\0\0\0\4\23\377\377\0
\0\0\24issuestatement\0\0\0\0\27\0\4\377\377\377\377resolution\0\0\0\0\27\0\4
\377\377\377\377own"..., 16384, 0) = 1448
-select(4, [3], [], [3], NULL)           = 1 (in [3])
-recv(3, "49:05+00\0\0\0\0321970-01-01 00:00:00+00\0\0\0\0322003-07-15
 14:58:29+00\0\0\0\0322003-07-16 14:30:24+00\0\0\0\0322003-08-13
 07:01:25+00\0\0\0\0051\0\0\0\0322003-08"..., 16271, 0) = 742

By comparison, the hanging version is, well, hung: nothing further is
emitted into the strace log and it just sits there until Ctrl-C is hit.

We are kind of up the creek here without being able to use the web UI,
even if we could live without ever searching by requestor.  Replying to
tickets using the command line interface and a Unix editor is less than
ideal.  (When doing that, is there any way to quote in the reply?)  I am
hoping that someone very familiar with the code will recognize this
problem as relatively simple and point me in the right direction for a
patch.  As explained, the Debian RT2 package stopped updating at 2.0.14
because the package maintainers switched focus to RT3, so upgrading to
2.0.15 would require pulling the system out from under package management,
which I am keen to avoid.

-- Mike