[Rt-devel] Encoding bug in ticket creation

[Andreas Vögele]

Jesse Vincent writes:

> On Aug 26, 2008, at 2:51 AM, Otmar Lendl wrote:
>
>> On 2008/08/15 11:08, Jesse Vincent <jesse at bestpractical.com> wrote:
>>>>
>>>> Aug 15 09:03:13 web1 RT: Attachment insert failed - ERROR:  invalid
>>>> byte sequence for encoding "UTF8": 0xf60a436f HINT:  [...]
>>>> (/opt/rt3/lib/RT/Attachment_Overlay.pm:153)
>>
>>>> I'm currently using RT 3.8.1rc5 with PostgreSQL 8.3.
>>> [...]
>> FYI, I'm seeing the same error.
>>
>> Here it's RT 3.7.21 / one of the RTIR milestone releases. The rest is
>> Debian stable, thus libdbd-pg-perl at 1.49-2, perl at 5.8.8-7etch3.
>>
>> In my case it's triggered by an incoming email, and I see the error in
>> the postgres logfile as well:
>>
>> 2008-08-26 11:25:07 CEST ERROR:  invalid byte sequence for encoding  
>> "UTF8": 0xe92064 [...]
>>
>> Drilling down on the email (It really pays to keep a copy of each
>> incoming mail) I found it labeled as:
>>
>> Content-Type: text/plain; charset="utf-8"
>> [...]
>> Now the kicker: the mail has a legal disclaimer appended (NOT
>> attached!), which containes 8-bit characters. Checking the
>> encoding shows that they are in latin1 and not utf-8.
>>
>> Great. In other words, the incoming mail is invalid. (removing the
>> disclaimer makes RT indeed happy.)
>>
>> Well, this is suboptimal. As a CERT, we can't just ignore slightly
>> broken mails. Any ideas?
>
> There's a switch to perl's UTF8 handling to replace broken utf8 in  
> text with blanks. Alternatively, QP could get forced before write to  
> the DB on sensitive databases.
>
> The routine you want to be looking at / fixing is lib/RT/Record.pm's  
> sub _EncodeLOB.
>
> A patch would be most welcome.

Has anybody patched this "bug" in the meanwhile?

One of my colleagues can't create tickets through the web interface if
his ticket subject contains umlaut characters.  He is able to insert
umlaut characters into the subject after he created the ticket though.
My colleague uses Firefox 3 and Opera on Ubuntu 8.10.  Both Firefox and
Opera fail.

Here's part of the error log file:

[Fri Nov 14 11:51:02 2008] [error] [client 192.168.0.1] FastCGI: server "/opt/rt3/bin/mason_handler.fcgi" stderr: DBD::Pg::st execute failed: ERROR:  invalid byte sequence for encoding "UTF8": 0xfc, referer: https://www.example.com/rt/Ticket/Create.html?Queue=6
[Fri Nov 14 11:51:02 2008] [error] [client 192.168.0.1] FastCGI: server "/opt/rt3/bin/mason_handler.fcgi" stderr: RT::Handle=HASH(0xa1ea8c8) couldn't execute the query 'INSERT INTO Attachments (ContentType, Parent, Subject, Headers, Creator, MessageId, Created, TransactionId) VALUES (?, ?, ?, ?, ?, ?, ?, ?)' at /usr/local/share/perl/5.8.8/DBIx/SearchBuilder/Handle.pm line 518, referer: https://www.example.com/rt/Ticket/Create.html?Queue=6
[Fri Nov 14 11:51:02 2008] [error] [client 192.168.0.1] FastCGI: server "/opt/rt3/bin/mason_handler.fcgi" stderr: \tDBIx::SearchBuilder::Handle::SimpleQuery('RT::Handle=HASH(0xa1ea8c8)', 'INSERT INTO Attachments (ContentType, Parent, Subject, Header...', 'text/plain', 0, 'Support f\xc3\xbcr Synapse 2008', 'Content-Type: text/plain; boundary="----------=_1226659862-10...', 40, '', '2008-11-14 10:51:02', ...) called at /usr/local/share/perl/5.8.8/DBIx/SearchBuilder/Handle.pm line 353, referer: https://www.example.com/rt/Ticket/Create.html?Queue=6

0xfc is the Latin-1 code for "ü".  \xc3\xbc is the byte representation
of "ü" in UTF-8.

I am able to reproduce this error on my colleague's Ubuntu machine, but
I couldn't reproduce it on other computers running OpenBSD, CentOS or
Windows.

Occasionally, the error "invalid byte sequence for encoding" is also
triggered by mail messages sent from iPhones.  It seems that the
iPhone's mail software mime-encodes the message subject but not other
mail headers like thread-topic.

I got several messages from iPhones where the subject and the body were
encoded in UTF-8 but where the thread-topic header contained non-encoded
Latin-1 characters.  Here's an example:

Content-Type: text/plain;
        format=flowed;
        delsp=yes;
        charset="utf-8"
Content-Transfer-Encoding: base64
thread-topic: [rt.example.com #1234] Bitte um Rückruf am Mittwoch
thread-index: AclI3xNq4KwWEYT8QX+ziU8EtI/A4g==
MIME-Version: 1.0 (iPhone Mail 5F136)
Subject: =?utf-8?B?UmU6[...]

The subject in this message contains an "ü" but I think that this
failure was triggered by the "ü" in the thread-topic header, since the
subject is properly mime-encoded.

[Mon Nov 17 19:05:42 2008] [error] [client 192.168.0.1] FastCGI: server "/opt/rt3/bin/mason_handler.fcgi" stderr: DBD::Pg::st execute failed: ERROR:  invalid byte sequence for encoding "UTF8": 0xfc
[Mon Nov 17 19:05:42 2008] [error] [client 192.168.0.1] FastCGI: server "/opt/rt3/bin/mason_handler.fcgi" stderr: RT::Handle=HASH(0x9f80578) couldn't execute the query 'INSERT INTO Attachments (ContentType, Parent, Subject, Filename, Headers, Creator, MessageId, Created, Content, ContentEncoding, TransactionId) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)' at /usr/local/share/perl/5.10.0/DBIx/SearchBuilder/Handle.pm line 518
[Mon Nov 17 19:05:42 2008] [error] [client 192.168.0.1] FastCGI: server "/opt/rt3/bin/mason_handler.fcgi" stderr: \tDBIx::SearchBuilder::Handle::SimpleQuery('RT::Handle=HASH(0x9f80578)', 'INSERT INTO Attachments (ContentType, Parent, Subject, Filena...', 'text/plain', 0, 'Re: [rt.example.com #1234] Bitte um R\\x{c3}\\x{bc}ckruf am Mitt...', undef, 'X-Amavis-Alert: BAD HEADER Non-encoded 8-bit data (char FC he...', 1366, '6FF0EA71-833A-4E91-8CD9-B2DF7AA22DE8 at example.com', ...) called at /usr/local/share/perl/5.10.0/DBIx/SearchBuilder/Handle.pm line 353

We use Request Tracker 3.8.1.  Our RT database instance runs under
Debian Lenny and PostgreSQL 8.3.4.  I upgraded the RT server yesterday
from Debian Etch to Lenny, which provides Perl 5.10 and libdbd-pg-perl
2.8.7, but that upgrade didn't change anything.

I'm currently reading Attachment_Overlay.pm but I'm not sure whether the
subject and the head are supposed to be encoded or decoded strings.
What kind of strings do the MIME::Head methods return?  And what kind of
strings do the database modules expect?

Regards,
Andreas