[rt-users] RT 4.4.1 - ExternalAuth intermittently failing

[Michel Daoust]

Hi,

My colleague Mike Johnson originally sent this request. I'd like to follow
up on our investigation.

We have figured out the why, but we still don't understand what is causing
this issue. Details below.

While debugging, we captured packets and received error code 52e from the
AD server (LDAP equivalent of error 49 RT was throwing). This indicates
that the login username is correct, but the password is incorrect. This is
what threw us off since it didn't make sense. The config user/pass is
correct and the account would work 90% of the time and we would get this
error seemingly at random.

So we decided to edit the RT source code where the LDAP bind happens
(/opt/rt4/lib/RT/Authen/ExternalAuth/LDAP.pm:634) and added lines to log
the $ldap_user and $ldap_pass variables to see what RT was trying to bind
with.

On successful binds, the username/password combination was correct, as
specified in our config. We waited for the LDAP bind error to happen and
when it did, we found this in the log:
ldap_user: ldap_user_account, ldap_pass:Password not printed. (literal
Password not Printed). This is whats causing the bind to fail. For some
reason, RT is sending the masked password string (used in the web ui when
looking at system configs), which of course isn't the right password for
the bind account.

As a temporary bandaid solution, we've hardcoded the LDAP bind credentials
in the source code. Since doing so, we haven't seen the error happen again.
So at this point, this seems like a bug in RT? Has anyone else experienced
something similar?

Thanks,

*Michel Daoust, BSc.*
Software Developer / Systems Analyst, Information Technology
Northern Ontario School of Medicine
935 Ramsey Lake Road
Sudbury, ON. P3E 2C6

Phone: 705-662-7193
Email: midaoust at nosm.ca


On Wed, Nov 23, 2016 at 3:21 PM, Mike Johnson <mike.johnson at nosm.ca> wrote:

> Hi,
>
> It happened again today. Our AD admin didn't see anything unusual in the
> logs. I'm getting him to see if successful bind attempts show up anywhere,
> and if so... if RT is actually successful and the error message is just not
> appropriate(ie something else behind the scenes is going on and it's just
> reported as a failed bind).
>
> Anyone have any thoughts on this?
>
> We have multiple other LDAP authenticated, and Windows authenticated
> systems on campus using this AD service(different usernames) and we haven't
> had reports of any of these failing.
>
> The things that have changed from what it was working:
> - OS: CentOS 7.2.15.11
> - perl 5.16.3
> - RT version 4.4.1
>
> I can't recall the previous OS version or perl version, but it was at
> least on Redhat 4 or 5, and RT was 3.8.X using ExternalAuth extension(on
> 3.8 it wasn't rolled into baseline yet).
>
> Any thoughts are appreciated!
> Mike.
>
> On Tue, Nov 22, 2016 at 4:40 PM, Kenneth Marshall <ktm at rice.edu> wrote:
>
>> On Tue, Nov 22, 2016 at 04:13:46PM -0500, Mike Johnson wrote:
>> > We just went live with RT 4.4.1 and it seems that LDAP authentication is
>> > failing.
>> >
>> > It has now died 2 days in a row, at approximately the same time.
>> >
>> > The RT log is showing the following error:
>> > 2819] [Mon Nov 21 21:10:28 2016] [critical]:
>> > RT::Authen::ExternalAuth::LDAP::_GetBoundLdapObj Can't bind:
>> > LDAP_INVALID_CREDENTIALS 49
>> > (/opt/rt4/sbin/../lib/RT/Authen/ExternalAuth/LDAP.pm:678)
>> >
>> > This seems like a generic LDAP error, and it's not pointing us to a
>> > specific issue.
>> >
>> > The user that we are binding with is a user that was in-use on our RT
>> 3.8.X
>> > environment that hadn't had an issue in years (3?).
>> >
>> > Restarting apache resolves the immediate issue, but clearly there is
>> > something else going on that we should be able to fix permanently.
>> Anyone
>> > have any ideas on where to look?
>> >
>> > This didn't come up in our testing, but I don't believe we had the
>> volume
>> > of credential testing that we do in prod.
>> >
>> > Any help would be great!
>> >
>> > P.S. The LDAP server is a Microsoft Active Directory server. This same
>> > server was being used for ExternalAuth extension in 3.8.
>> >
>> > Mike.
>>
>> Hi Mike,
>>
>> You probably will need to check your AD logs as well. We have seen issues
>> with AD authentication when an account is locked out by a bad password
>> login attempt. Since it is about the same time of day, maybe something
>> else is trying to login with those credentials and causing it to lock.
>>
>> Regards,
>> Ken
>>
>
>
>
> --
> Mike Johnson
> Datatel Programmer/Analyst
> Northern Ontario School of Medicine
> 955 Oliver Road
> Thunder Bay, ON   P7B 5E1
> Phone: (807) 766-7331
> Email: mike.johnson at nosm.ca
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bestpractical.com/pipermail/rt-users/attachments/20161202/d303e86e/attachment.htm>