[rt-users] Bad characters in names loaded from LDAP (AD)

[Bill Cole]

On 11 Oct 2016, at 5:51, Jan Burian wrote:

> Hi Bill,
>
> thank you for your response. Sry not to mention our database.
> We use PostreSQL.
> After I wrote first email a also checked encoding in database.
>
> The database was with following parameters:
>    Name    | Encoding |      Collate     |       Ctype
> -------------+-------------+-----------------+------------------
>       rt4      |  UTF8       | en_US.UTF-8 | en_US.UTF-8

And so my beautiful theory is destroyed by your brutal facts. :)

> 1) I dump database with UTF-8 encoding parameter.
> 2) Then I drop the databases.
> 3) Create new database with following parameters:
>
>    Name    | Encoding |      Collate     |       Ctype
> -------------+-------------+-----------------+------------------
>       rt4      |  UTF8       | cs_CZ.UTF-8 | cs_CZ.UTF-8
>
> 4) And then import database from dump.
>
> But after that change names are loading from LDAP still with bad
> characters :-/.

Indeed: the Collate and Ctype parameters are encoding-specific rulesets 
for how characters are related to each other, not variations on 
encoding.

> When the user writes first email to queue, then is also autocreated as
> unprivileged. If he/she was his/her name in From header, then is used 
> as
> RealName RT attribute. But in this case is his/her name saved 
> correctly.
>
> *Example from the log - autocreated from LDAP:*
> [6937] [Tue Sep 27 15:59:25 2016] [info]:
> RT::User::CanonicalizeUserInfoFromExternalAuth returning Disabled: ,
> EmailAddress: novak at vsup.cz, Gecos: novak, Name: novak, Privileged: 1,
> RealName: Matouš Novák, WorkPhone:  
> (/opt/rt4/sbin/../lib/RT/User.pm:811)
> [6937] [Tue Sep 27 15:59:25 2016] [info]: Autocreated external user
> novak ( 61 ) (/opt/rt4/sbin/../lib/RT/Authen/ExternalAuth.pm:356)
> [6937] [Tue Sep 27 15:59:25 2016] [info]:
> RT::Authen::ExternalAuth::LDAP::GetAuth External Auth OK ( My_LDAP ):
> novak (/opt/rt4/sbin/../lib/RT/Authen/ExternalAuth/LDAP.pm:348)
> [6937] [Tue Sep 27 15:59:26 2016] [info]:
> RT::User::CanonicalizeUserInfoFromExternalAuth returning EmailAddress:
> novak at vsup.cz, Name: novak, *RealName: Matouš Novák*, WorkPhone:
> (/opt/rt4/sbin/../lib/RT/User.pm:811)
> *
> **Example from the log - autocreated from email:*
> [6026] [Mon Oct 10 06:26:02 2016] [info]:
> RT::User::CanonicalizeUserInfoFromExternalAuth returning Comments:
> Autocreated on ticket submission, Disabled: , EmailAddress:
> Tereza.Skvarova at seznam.cz, Name: Tereza.Skvarova at seznam.cz, 
> Privileged:
> , *RealName: Tereza Škvárová* (/opt/rt4/sbin/../lib/RT/User.pm:811)
>
> Any other ideas?

Yes: At least one of your FCGI handlers (PID 6937) is using an 8-bit 
encoding and at least one (PID 6026) is using UTF-8.

Note that both of those cases are being logged by the 
RT::User::CanonicalizeUserInfoFromExternalAuth method, which uses LDAP 
to retrieve the attribute it uses for the "RealName" field in RT. The 
first was logged by process 6937, the second by process 6026.

The *reason* for that is a bit of a mystery. It's clear that the 2 
processes were not started near the same time (unless that server is 
VERY busy spawning processes) so if you can determine what was different 
about how they were launched (likely a involving a locale environment 
variable, most likely LANG or LC_ALL) you can probably make sure that 
the improper launch doesn't happen.