[Rt-commit] rt branch 5.0/rt-importer-reduce-memory created. rt-5.0.1-609-ge185137a78

[BPS Git Server]

This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "rt".

The branch, 5.0/rt-importer-reduce-memory has been created
        at  e185137a787acd3910cde082793d0ee5972c84cf (commit)

- Log -----------------------------------------------------------------
commit e185137a787acd3910cde082793d0ee5972c84cf
Author: sunnavy <sunnavy at bestpractical.com>
Date:   Sat Sep 4 04:09:10 2021 +0800

    Reduce memory usage for rt-importer
    
    We keep track of all created objects' UID and related class/id in a hash
    during the whole process, the structure was like:
    
        {
            UID => [ CLASS, ID ],
            UID2 => [ CLASS2, ID2 ],
        }
    
    This is not quite efficient and could consume a lot of memory when there
    are millions of UIDs. This commit refactors the structure to somethig
    like:
    
        {
            UID => CLASS-ID,
            UID2 => undef,
        }
    
    For the value part, replacing arrayrefs with strings reduces the memory
    about 50%. Furthermore, when we can infer class/id from uid(which is the
    case when importing from "rt-serializer --clone"), there is no need to
    store strings either, and replacing strings with undef can reduce memory
    about 20%.

diff --git a/lib/RT/Migrate/Importer.pm b/lib/RT/Migrate/Importer.pm
index 061d8a402c..79f14ccdac 100644
--- a/lib/RT/Migrate/Importer.pm
+++ b/lib/RT/Migrate/Importer.pm
@@ -163,7 +163,15 @@ sub InitStream {
 sub Resolve {
     my $self = shift;
     my ($uid, $class, $id) = @_;
-    $self->{UIDs}{$uid} = [ $class, $id ];
+
+    # If we can infer from uid, do not store class/id to save memory usage.
+    if ( $uid eq join '-', $class, $self->{Organization}, $id ) {
+        $self->{UIDs}{$uid} = undef;
+    }
+    else {
+        $self->{UIDs}{$uid} = "$class-$id";
+    }
+
     return unless $self->{Pending}{$uid};
 
     for my $ref (@{$self->{Pending}{$uid}}) {
@@ -196,7 +204,15 @@ sub Lookup {
         carp "Tried to lookup an undefined UID";
         return;
     }
-    return $self->{UIDs}{$uid};
+
+    return unless exists $self->{UIDs}{$uid};
+
+    if ( ( $self->{UIDs}{$uid} // '' ) =~ /(.+)-(.+)/
+        || $uid =~ /(.+)-(?:\Q$self->{Organization}\E)-(.+)/ )
+    {
+        return [ $1, $2 ];
+    }
+    return;
 }
 
 sub LookupObj {

-----------------------------------------------------------------------


hooks/post-receive
-- 
rt