[rt-devel] Spot the difference (attachment corruption)

[Iain Price]

This one is good, i hope someone has experience and can just say 'oh 
yeah do this' :)

Having deployed rt3.0.5 i was unhappy to find my users couldn't do 
attachments properly without corruption - especially as i had spent so 
long testing it on my dev box.  Anyway after several days of chasing 
dead ends i found out two things
1) my 'dev' box isn't the same as my live box for some reason, the dev 
box which works properly is running RH 8 while the live box is running 
RH 9 (and smashes up attachments)
2) the corruption consists of escaping top bit set chars (see below)
additionally to note
3) both boxes are running the same copy of apache 1.3 manually compiled
4) both boxes are running the same copy of mod_perl 1.26 manually compiled
5) same rt too, and both pass the rt-testdependancies --with-mysql 
--with-modperl1 the same way - except
6) Digest::MD5 on both boxes is out of date according to testdeps.  I 
doubt this is the issue here, but the newer versions (the one requested 
in testdeps and later ones) produce junk for a Makefile when i do the 
usual perl Makefile.PL - maybe i did something wrong but the Makefile 
has serious syntax errors, invalid labels, unterminated strings, missing 
values for attributes, something very wierd here - and on both RH8 and 
RH9 box :P

so the difference is presumably in the core os packages.   since apache 
and mod_perl are manually compiled, i'm looking over at you Mr Perl...
Well i dont know, thats kinda why i'm here, is it a core library maybe, 
perl its self (both machines have 5.8.0 tho) or maybe some module that 
perl uses thats not currently checked fully against testdeps, or 
something thats maybe /too new/ for RT to understand.  Or something.  I 
dunno, any suggestions?

Anyway here is the hex source of my test file (from "od -tx1 filename")
0000000 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f
0000020 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f
0000040 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f
0000060 30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d 3e 3f
0000100 40 41 42 43 44 45 46 47 48 49 4a 4b 4c 4d 4e 4f
0000120 50 51 52 53 54 55 56 57 58 59 5a 5b 5c 5d 5e 5f
0000140 60 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f
0000160 70 71 72 73 74 75 76 77 78 79 7a 7b 7c 7d 7e 7f
0000200 80 81 82 83 84 85 86 87 88 89 8a 8b 8c 8d 8e 8f
0000220 90 91 92 93 94 95 96 97 98 99 9a 9b 9c 9d 9e 9f
0000240 a0 a1 a2 a3 a4 a5 a6 a7 a8 a9 aa ab ac ad ae af
0000260 b0 b1 b2 b3 b4 b5 b6 b7 b8 b9 ba bb bc bd be bf
0000300 c0 c1 c2 c3 c4 c5 c6 c7 c8 c9 ca cb cc cd ce cf
0000320 d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 da db dc dd de df
0000340 e0 e1 e2 e3 e4 e5 e6 e7 e8 e9 ea eb ec ed ee ef
0000360 f0 f1 f2 f3 f4 f5 f6 f7 f8 f9 fa fb fc fd fe ff
0000400

creative huh

and heres the results from downloading it from rt3 (its shown in the 
attachments list as 356 bytes, exactly the expected 128 too many, and 
even the stored form in the DB is broken)

0000000 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f
0000020 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f
0000040 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f
0000060 30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d 3e 3f
0000100 40 41 42 43 44 45 46 47 48 49 4a 4b 4c 4d 4e 4f
0000120 50 51 52 53 54 55 56 57 58 59 5a 5b 5c 5d 5e 5f
0000140 60 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f
0000160 70 71 72 73 74 75 76 77 78 79 7a 7b 7c 7d 7e 7f
0000200 c2 80 c2 81 c2 82 c2 83 c2 84 c2 85 c2 86 c2 87
0000220 c2 88 c2 89 c2 8a c2 8b c2 8c c2 8d c2 8e c2 8f
0000240 c2 90 c2 91 c2 92 c2 93 c2 94 c2 95 c2 96 c2 97
0000260 c2 98 c2 99 c2 9a c2 9b c2 9c c2 9d c2 9e c2 9f
0000300 c2 a0 c2 a1 c2 a2 c2 a3 c2 a4 c2 a5 c2 a6 c2 a7
0000320 c2 a8 c2 a9 c2 aa c2 ab c2 ac c2 ad c2 ae c2 af
0000340 c2 b0 c2 b1 c2 b2 c2 b3 c2 b4 c2 b5 c2 b6 c2 b7
0000360 c2 b8 c2 b9 c2 ba c2 bb c2 bc c2 bd c2 be c2 bf
0000400 c3 80 c3 81 c3 82 c3 83 c3 84 c3 85 c3 86 c3 87
0000420 c3 88 c3 89 c3 8a c3 8b c3 8c c3 8d c3 8e c3 8f
0000440 c3 90 c3 91 c3 92 c3 93 c3 94 c3 95 c3 96 c3 97
0000460 c3 98 c3 99 c3 9a c3 9b c3 9c c3 9d c3 9e c3 9f
0000500 c3 a0 c3 a1 c3 a2 c3 a3 c3 a4 c3 a5 c3 a6 c3 a7
0000520 c3 a8 c3 a9 c3 aa c3 ab c3 ac c3 ad c3 ae c3 af
0000540 c3 b0 c3 b1 c3 b2 c3 b3 c3 b4 c3 b5 c3 b6 c3 b7
0000560 c3 b8 c3 b9 c3 ba c3 bb c3 bc c3 bd c3 be c3 bf

seems to be an obsession with 0xC2 escaping on chars 0x80-0xbf and 0xc3 
for 0xc0-0xff but the values are 0x30 less than they should be?
i'm not familiar with this encoding type :)
I'm only guessing but... bytes 0xc0-0xff are reserved for selecting some 
kind of character map which overlays 64 chars onto 0x80-0xbf, presumably 
maps 0xc0 to 0xc3 store the entire ascii set in 4 blocks - 0-63, 64-127, 
128-195, 196-255, with the bonus that blocks 0xc0 and 0xc1 aren't 
necessary as that part of the map isn't overwritten... optimised for 
ASCII-7 coding.... heck i dont know, its probably some kind of standard 
encoding, i'm just guessing how it might work and what it might be from 
the above data, i could easily be completely wrong and barking up the 
wrong tree :)



Iain