[rt-users] Encoding Troubles with RT 3.8.8

[benoit plessis]

Hi,

I've some problems with iso-8859-1/utf8 encoding of outbound mail with RT.

Exemple: Email received in 'iso-8859-1', is correctly displayed in
UTF-8 within RT,
when showing 'Full Header' we have on top:
  content-type: text/plain; charset="utf-8"

and on bottom:
  X-RT-Original-Encoding: ISO-8859-1

If we forward this email to a new address, the forwarded message
include the same headers, however the message body is in "quoted
printable" encoding of a iso-8859-1 message:

==Extract of forwarded email: ======================================
This is a forward of ticket #260

------------=_1280925536-13250-3
Content-Type: message/rfc822
Content-Disposition: attachment
Content-Transfer-Encoding: binary
Content-Description: forwarded message

Content-Type: multipart/mixed; boundary="----------=_1280925536-13250-2"
Content-Transfer-Encoding: binary
MIME-Version: 1.0
X-Mailer: MIME-tools 5.427 (Entity 5.427)

This is a multi-part message in MIME format...

------------=_1280925536-13250-2
MIME-Version: 1.0
X-RT-Queue: Support
Content-Type: text/plain; charset="utf-8"
....
X-RT-Original-Encoding: ISO-8859-1
Content-Length: 1116

Bonjour,

Je fais suite =E0 notre entretien t=E9l=E9phonique.
....
==Extract of forwarded email: ======================================



Also, there is a similar problem when using html mail/template,
accentuated chars are converted to iso-8859-1 chars for the
"text/plain" version of the mail with is send with an utf8 headers.
The text/html part contain the html-entity codes so there is no problem.

When not using html templates entities are correctly converted using
utf8 charset.

For this part, in _DowngradeFromHTML there is two thing:
 * forced 'utf8' mime charset, why not using "EmailOutputEncoding"
 * use of HTML::FormatText which can only generate latin1 text


   my $charset = RT->Config->Get('EmailOutputEncoding');

   my $new_entity = $orig_entity->dup; # this will fail badly if we go
away from InCore parsing
   $new_entity->head->mime_attr( "Content-Type" => 'text/plain' );
   $new_entity->head->mime_attr( "Content-Type.charset" => $charset );

   $orig_entity->head->mime_attr( "Content-Type" => 'text/html' );
   $orig_entity->head->mime_attr( "Content-Type.charset" => $charset );
   $orig_entity->make_multipart('alternative', Force => 1);

   require HTML::FormatText;
   require HTML::TreeBuilder;
   require Encode;
   my $tree = HTML::TreeBuilder->new_from_content(
       $new_entity->bodyhandle->as_string
   );
   $new_entity->bodyhandle(MIME::Body::InCore->new(
       \(
        Encode::encode($charset,
                Encode::decode("iso-8859-1",
                        scalar HTML::FormatText->new(
                           leftmargin  => 0,
                           rightmargin => 78,
                       )->format( $tree )
                )
       )
   )
);
   $tree->delete;
   ....