[rt-devel] RT 4.4 slow sbin/rt-externalize-attachments with large and old (before RT 4.2) databases

[Christian Loos]

Hi,

I'm currently upgrading one of our old RT instances from 3.8.1 to 4.4.0.
The instance started with RT 3.4.2 and has over 7,000,000 attachments.

sbin/rt-externalize-attachments is really slow, 1,500,000 attachments in
4 hours.

The MySQL processlist shows me always 'UPDATE Attachments SET Headers='
queries.
This is because rt-externalize-attachments calls
RT::Attachment->ContentLength [1] which checks if a Content-Length
header exist, and if not, update the header with a Content-Length header
value.

Unfortunately RT never set a Content-Length header for created
attachments before 4.2.0 [2]. While incoming mails and attachments
expected to have a Content-Length header, attachments created by RT
themselves (ticket create by WebUI/REST, OutgoingEmailRecord) doesn't
have one.

My question:
is it really necessary that rt-externalize-attachments update all
attachment which doesn't have a Content-Length header value?

I already moved attachments from one of our RT 4.2. instances out of the
database with RT::Extension::ExternalStorage, which fetches the content
length from the DB [3], which is much faster (1,500,000 attachments in 2
hours).

Chris

[1]
https://github.com/bestpractical/rt/blob/stable/lib/RT/Attachment.pm#L426-L439
[2]
https://github.com/bestpractical/rt/commit/337893c1f5b53a430667107b28b8212acdaf037e
[3]
https://github.com/bestpractical/rt-extension-externalstorage/blob/master/sbin/extract-attachments.in#L99-L105