[rt-devel] Debug info on international searches

[Ruslan U. Zakirov]

	Hello, Jesse.
Today I debug nasty searches with international searches.

1) If I drop UTF encode of ARGS in autohandler then searches work.
2) RT(more Mason, more more HTML::Entities) do wrong escaping of perl 
UTF encoded strings. For eg: it's do %HHH instead of %HH%HH for Russian 
chars. It could be fixed with own escaping:

$m->interp->set_escape( rtu =>
sub {
         return unless defined ${ $_[0] };
         require Encode;
         Encode::_utf8_off( ${ $_[0] } );
         ${ $_[0] } =~ s/([^a-zA-Z0-9_.-])/uc sprintf("%%%02x",ord($1))/eg;

});

If we don't fix this then browser intrpret this wrong and sends back 
wrong sequence.

3) This not fix all problems, because we don't have unescaping of such 
sequences.
I nasty implement unescaping in autohandler before UTF encoding just for 
test:

%ARGS = map {
     # if they've passed multiple values, they'll be an array. if they've
     # passed just one, a scalar whatever they are, mark them as utf8
     my $type = ref($_);
     (!$type)
         ? s/%([0-9A-Fa-f]{2})/chr(hex($1))/eg? $_ : $_ :
         ($type eq 'ARRAY')
         ? [ map { ref($_) ? $_ : s/%([0-9A-Fa-f]{2})/chr(hex($1))/eg? 
$_ : $_ } @$_ ] :
         ($type eq 'HASH')
         ? { map { ref($_) ? $_ : s/%([0-9A-Fa-f]{2})/chr(hex($1))/eg? 
$_ : $_ } %$_ } : $_
     } %ARGS;

This fix all FreezeThaw problems.

4) But searches don't work still because something escape high bit chars.
Example of Dumped $TicketsObj->{TicketRestrictions}
$VAR1 = {
           '14' => {
                     'FIELD' => 'Subject',
                     'VALUE' => "\x{442}\x{435}\x{441}\x{442}",
                     'OPERATOR' => 'LIKE',
                     'DESCRIPTION' => "Subject LIKE 
\x{442}\x{435}\x{441}\x{442}"
                   }
         };

MySQL don't like it or may be something else. When ARGS was not encoded 
everything was fine with search, but it's not right fix.

Have to go home right now.
			Best regards, Ruslan.