<div dir="ltr"><div class="gmail_extra"><div><div class="gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hi,</div><div><br></div><div>My colleague Mike Johnson originally sent this request. I'd like to follow up on our investigation.</div><div><br></div><div>We have figured out the why, but we still don't understand what is causing this issue. Details below.</div><div><br></div><div>While debugging, we captured packets and received error code 52e from the AD server (LDAP equivalent of error 49 RT was throwing). This indicates that the login username is correct, but the password is incorrect. This is what threw us off since it didn't make sense. The config user/pass is correct and the account would work 90% of the time and we would get this error seemingly at random.</div><div><br></div><div>So we decided to edit the RT source code where the LDAP bind happens (/opt/rt4/lib/RT/Authen/ExternalAuth/LDAP.pm:634) and added lines to log the $ldap_user and $ldap_pass variables to see what RT was trying to bind with.</div><div><br></div><div>On successful binds, the username/password combination was correct, as specified in our config. We waited for the LDAP bind error to happen and when it did, we found this in the log:</div><div>ldap_user: ldap_user_account, ldap_pass:Password not printed. (literal Password not Printed). This is whats causing the bind to fail. For some reason, RT is sending the masked password string (used in the web ui when looking at system configs), which of course isn't the right password for the bind account.</div><div><br></div><div>As a temporary bandaid solution, we've hardcoded the LDAP bind credentials in the source code. Since doing so, we haven't seen the error happen again. So at this point, this seems like a bug in RT? Has anyone else experienced something similar?</div><div><br></div><div>Thanks,</div><div><br></div><div><div><b style="font-size:12.8px">Michel Daoust, BSc.</b><br></div><div>Software Developer / Systems Analyst,<span style="font-size:12.8px"> Information Technology</span></div><div>Northern Ontario School of Medicine</div><div>935 Ramsey Lake Road</div><div>Sudbury, ON. P3E 2C6</div><div><br></div><div>Phone: 705-662-7193</div><div>Email: <a href="mailto:midaoust@nosm.ca" target="_blank">midaoust@nosm.ca</a></div></div><div><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Wed, Nov 23, 2016 at 3:21 PM, Mike Johnson <span dir="ltr"><<a href="mailto:mike.johnson@nosm.ca" target="_blank">mike.johnson@nosm.ca</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>It happened again today. Our AD admin didn't see anything unusual in the logs. I'm getting him to see if successful bind attempts show up anywhere, and if so... if RT is actually successful and the error message is just not appropriate(ie something else behind the scenes is going on and it's just reported as a failed bind).</div><div><br></div><div>Anyone have any thoughts on this?<br></div><div><br></div><div>We have multiple other LDAP authenticated, and Windows authenticated systems on campus using this AD service(different usernames) and we haven't had reports of any of these failing.</div><div><br></div><div>The things that have changed from what it was working:</div><div>- OS: CentOS 7.2.15.11</div><div>- perl 5.16.3</div><div>- RT version 4.4.1<br></div><div><br></div><div>I can't recall the previous OS version or perl version, but it was at least on Redhat 4 or 5, and RT was 3.8.X using ExternalAuth extension(on 3.8 it wasn't rolled into baseline yet).</div><div><br></div><div>Any thoughts are appreciated!</div><div>Mike.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Nov 22, 2016 at 4:40 PM, Kenneth Marshall <span dir="ltr"><<a href="mailto:ktm@rice.edu" target="_blank">ktm@rice.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span>On Tue, Nov 22, 2016 at 04:13:46PM -0500, Mike Johnson wrote:<br>
> We just went live with RT 4.4.1 and it seems that LDAP authentication is<br>
> failing.<br>
><br>
> It has now died 2 days in a row, at approximately the same time.<br>
><br>
> The RT log is showing the following error:<br>
> 2819] [Mon Nov 21 21:10:28 2016] [critical]:<br>
> RT::Authen::ExternalAuth::LDAP<wbr>::_GetBoundLdapObj Can't bind:<br>
> LDAP_INVALID_CREDENTIALS 49<br>
> (/opt/rt4/sbin/../lib/RT/Authe<wbr>n/ExternalAuth/LDAP.pm:678)<br>
><br>
> This seems like a generic LDAP error, and it's not pointing us to a<br>
> specific issue.<br>
><br>
> The user that we are binding with is a user that was in-use on our RT 3.8.X<br>
> environment that hadn't had an issue in years (3?).<br>
><br>
> Restarting apache resolves the immediate issue, but clearly there is<br>
> something else going on that we should be able to fix permanently. Anyone<br>
> have any ideas on where to look?<br>
><br>
> This didn't come up in our testing, but I don't believe we had the volume<br>
> of credential testing that we do in prod.<br>
><br>
> Any help would be great!<br>
><br>
> P.S. The LDAP server is a Microsoft Active Directory server. This same<br>
> server was being used for ExternalAuth extension in 3.8.<br>
><br>
> Mike.<br>
<br>
</span>Hi Mike,<br>
<br>
You probably will need to check your AD logs as well. We have seen issues<br>
with AD authentication when an account is locked out by a bad password<br>
login attempt. Since it is about the same time of day, maybe something<br>
else is trying to login with those credentials and causing it to lock.<br>
<br>
Regards,<br>
Ken<span class="gmail-HOEnZb"><font color="#888888"><br>
</font></span></blockquote></div><span class="gmail-HOEnZb"><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div class="gmail-m_-6145814284715752659gmail_signature">Mike Johnson<br>Datatel Programmer/Analyst<br>Northern Ontario School of Medicine<br>955 Oliver Road<br>Thunder Bay, ON P7B 5E1<br>Phone: <a href="tel:(807)%20766-7331" value="+18077667331" target="_blank">(807) 766-7331</a><br>Email: <a href="mailto:mike.johnson@nosm.ca" target="_blank">mike.johnson@nosm.ca</a></div>
</font></span></div>
</blockquote></div><br></div></div>