[rt-devel] RT::Template not unfolding long header lines

[Bruce Campbell]

	Was Re: [rt-users] bad rfc822 field name? (fwd)

On Wed, 28 Nov 2001, matt carter wrote:

> i have done some further investigation of my problem and it would appear
> that pine may be the culprit. i was hoping some other people may have an
> opinion on this, and an appropriate way to deal with.
>
> my suggestion would be to check for presence of : on the line before we
> assume its part of a new header line?

Looking at this old problem (which we're getting at the moment, which
throws mh for a merry dance), it seems to be in RT::Template being rather
silly about headers ( and I'm a tad busy to go and force Jesse to write 'I
will not write a mail-parser without understanding mail standards' several
thousand times ;) )

RFC822 cites:

     3.1.1.  LONG HEADER FIELDS

        Each header field can be viewed as a single, logical  line  of
        ASCII  characters,  comprising  a field-name and a field-body.
        For convenience, the field-body  portion  of  this  conceptual
        entity  can be split into a multiple-line representation; this
        is called "folding".  The general rule is that wherever  there
        may  be  linear-white-space  (NOT  simply  LWSP-chars), a CRLF
        immediately followed by AT LEAST one LWSP-char may instead  be
        inserted.  Thus, the single line

            To:  "Joe & J. Harvey" <ddd @Org>, JJV @ BBN

        can be represented as:

            To:  "Joe & J. Harvey" <ddd @ Org>,
                    JJV at BBN

[more examples]

             The process of moving  from  this  folded   multiple-line
        representation  of a header field to its single line represen-
        tation is called "unfolding".  Unfolding  is  accomplished  by
        regarding   CRLF   immediately  followed  by  a  LWSP-char  as
        equivalent to the LWSP-char.

So, pulling a sample out of our hat, we have a multi-line subject header
as follows:

	Subject: here is a sample of SPAM Filth and Depravity that I have
	  received either, from your service, from a service that you host,
	  solicitoring replies to your service. In my country, this kind of
	  email is illegal. please investigate.

Line 292 of RT::Template builds up the MIME::Entity based on a
split(/\n/).  Ok, but theres no joining of header lines that have been
folded.

So, if we change the loop to read:

  if ($headers) {
    my $prevkey = undef;
    foreach $header (split(/\n/,$headers)) {
      if( $header =~ /^\s+/ && defined( $prevkey) ){
        # This is a continuation.
        # Although implied in the docs, "-1" as the index to add doesn't
        # seem to work as wanted.  Use replace.
        $self->{'MIMEObj'}->head->add( $prevkey, $self->{'MIMEObj'}->head->get( $prevkey ) . $header );
        # Note that technically we should be doing get($prevkey) . "\r\n" . $header
        # and then calling unfold() as the above could lead to strange
	# numbers of spaces.
      }else{
        (my $key, my $value) = (split(/: /,$header,2));
        chomp $key;
        chomp $value;
        $self->{'MIMEObj'}->head->fold_length($key,10000);
        $self->{'MIMEObj'}->head->add($key, $value);
        # Cache the key in case it continues.
        $prevkey = $key;
      }
    }
    # Unfold all the headers if still needed.
    $self->{'MIMEObj'}->head->unfold();
  }

Applying this change to RT::Template results no more complaints about bad
RFC822 headers ;)

Other portions of the RT code base also exhibit certain confusion about
folded headers.

-- 
                             Bruce Campbell                            RIPE
                                    RIPE 41                             NCC
                       Jan 14-18, Amsterdam                      Operations