[Rt-devel] Norwegian, Swedish and Danish character problems..

[Hans K. Rosbach]

We have a single server set up for our multi-country rt3 server.
But we have a big problem..  I regard it as a showstopper, due to
the amount of these letters beeing used in the affected languages.

The characters we are having trouble with are:
ÆØÅ æøå ÅÄÖ åäö  (And I do not know the Danish ones)

All of these appear as double-character codes. ex:  æ
This does not happen always, but I would say something like 50%
of the new messages that arrive are like this.

Now, it seems to me like the EmailInputEncodings setting in the
config file is somehow fault, because changing the order of the types
does change what messages appear wrong and what are right. It
actually seems to me like rt only sees the first one, then uses that no
matter what the actual format of the mail is. Or something..

>From my RT conf file:
 @EmailInputEncodings = qw(utf-8 iso-8859-1 us-ascii) unless
(@EmailInputEncodings);
 Set($EmailOutputEncoding , 'utf-8');

Apache has been set to use UTF-8 as default.

>From one of the tickets:
  content-type: text/plain; charset="utf-8"
  X-RT-Original-Encoding: iso-8859-1

  Ønsker å bli kontaktet angående

This is clearly wrong.. And this is really a big problem.

I have checked, and it is not true that this only affects messages
written in Outlook.
I have seen this problem appear with:

 -PHP's mail(); function using the same server as rt is hosted on
        content-type: text/plain; charset="utf-8"
        X-RT-Original-Encoding: iso-8859-1

 -Apple Mail (2.613):
         content-type: text/plain; charset="utf-8"; format="flowed"
         X-RT-Original-Encoding: ISO-8859-1

 -Outlook Express:
         content-type: text/plain; charset="utf-8"
         X-RT-Original-Encoding: iso-8859-1

 -Microsoft something ( MimeOLE V6.00.2800.1165)
         content-type: text/plain; charset="utf-8"
         X-RT-Original-Encoding: us-ascii

So, to me it seems like it somehow converts the message into utf-8,
but forgets to save to the ticket info that it has been converted. Thus
converts it again for output. Because.. none of these messages were
sent using utf-8 type.

Sending an original utf-8 message (from Evolution) to rt works just
fine.. But I cannot make everyone change to Evolution either..

I hope this helps in finding the cause..

I run:
RT 3.0.10
perl-HTML-Parser-3.26-17
perl-XML-Parser-2.31-15
perl-libxml-enno-1.02-29
perl-DBD-MySQL-2.1021-3
perl-suidperl-5.8.0-88.3
perl-GD-Text-Util-0.86-0.rh90.dag
perl-Filter-1.29-3
perl-HTML-Tagset-3.03-28
perl-Parse-Yapp-1.05-30
perl-libwww-perl-5.65-6
perl-libxml-perl-0.07-28
perl-XML-Encoding-1.01-23
perl-XML-Grove-0.46alpha-25
perl-DBI-1.32-5
perl-GD-1.41-1.rh90.dag
perl-GD-Graph-1.43-1
perl-5.8.0-88.3
perl-CGI-2.81-88.3
perl-DateManip-5.40-30
perl-URI-1.21-7
perl-XML-Dumper-0.4-25
perl-XML-Twig-3.09-3
perl-CPAN-1.61-88.3
Apache 2.0.47 w/FastCGI

-HK