Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 101157
Summary: | Errata packages cause slow login and bogus log entry | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | [Retired] Red Hat Linux | Reporter: | Jordan Russell <jr-redhatbugs2> | ||||||
Component: | openssh | Assignee: | Nalin Dahyabhai <nalin> | ||||||
Status: | CLOSED NOTABUG | QA Contact: | Brian Brock <bbrock> | ||||||
Severity: | medium | Docs Contact: | |||||||
Priority: | high | ||||||||
Version: | 9 | CC: | arkadius.branczyk, brian, disposablehero3000-redhat, dmircea, jcaruso, jrojas, kasparek, kearns, kevin_myer, kmoir, menscher, michael.houle, mike, milan.kerslager, mitr, petr, psr, redhatbugzillaold, reuben-redhatbugzilla, rhbugzilla, sgarner, shishz, starback | ||||||
Target Milestone: | --- | ||||||||
Target Release: | --- | ||||||||
Hardware: | i386 | ||||||||
OS: | Linux | ||||||||
Whiteboard: | |||||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||||
Doc Text: | Story Points: | --- | |||||||
Clone Of: | Environment: | ||||||||
Last Closed: | 2003-08-05 06:27:50 UTC | Type: | --- | ||||||
Regression: | --- | Mount Type: | --- | ||||||
Documentation: | --- | CRM: | |||||||
Verified Versions: | Category: | --- | |||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||
Embargoed: | |||||||||
Attachments: |
|
Description
Jordan Russell
2003-07-29 18:07:21 UTC
Some other notes: 1. In my example, I was using RSA authentication with SSH protocol version 2, but I can reproduce the same issues with password authentication. 2. The issues are reproducable with the stock /etc/ssh/sshd_config file. 3. I'm not using any non-standard authentication scheme. 4. I can also reproduce the issues with a clean install of Red Hat Linux 9.0, with only the openssh errata packages installed. (My other boxes are upgrades from prior Red Hat Linux versions, and they have all errata packages applied.) If you don't need password authentication you can put in: PasswordAuthentication no into /etc/ssh/sshd_config. This seems to skip the bogus pam authentication request (which avoids the bogus log entry and the delay). But of course this doesn't help if you need password authentication. The same behavior can be seen on RH7.3 after last openssh update (openssh-3.1p1-8.i386.rpm) The correct fix for this is probably patch PAM to produce same delay when accept bad login-name or bad password with correct login-name. Patching ssh this way is wrong as there could be another service that will have the same possibility to see what account is wrong and what is not. Actually, I get the same thing going from a severn machine to another severn machine, with stock packages... real 0m5.039s user 0m0.140s sys 0m0.000s This should be marked a duplicate of 101361 and marked critical. The patch is simply incorrect and should not be used. It addresses only a minor brute-force user enumeration non-problem. Bug #101361 is not duplicate. This bug is another story about Kerberos but the next bad behavior of this broken security update. Actually, this is what the fix is expected to do. OpenSSH would skip PAM authentication for user accounts which didn't exist or for passwords which didn't meet its internal requirements (no empty passwords, and so on). When PAM authentication fails, libpam, as configured, forces a delay. OpenSSH, by short-circuiting the PAM authentication check, leaks information about whether or not an account exists for a given user through the length of the delay. The fix implemented in 3.6.1 and backported to these errata packages is to *always* attempt PAM authentication, even if the account or password fails OpenSSH's internal checks (though if the authentication attempt has already failed sshd's internal checks, regardless of whether or not libpam says "ok", the authentication attempt will fail). I find a full 2 second delay for *successful logins* unacceptable. Likewise - to a lesser degree - "authentication failure" log messages for successful logins. I find it very hard to believe that both of these side effects were 100% intentional, and further that they can't be worked around. Therefore, I can't help but disagree with your shrugging this off as "not a bug", and hope that you reconsider your stance. The majority seem to be believe that either this "fix" should be reverted, or that more work needs to be done on it. As of now, I'm back using previous openssh packages, and apparently others here are as well. If this is not fixed, it's going to create a sticky situation when a "real" security issue is discovered in openssh and we're not able to upgrade. I agree with JR that this "fix" seems ridiculous. Why should I have login delays and generate log messages when there have been no login failures?!? We have a centralized location where all syslog entries on our network are sent. We have a periodic cron job that evaluates the common syslog and sends three system administrators e-mail with items of interest that should be investigated. Now we get all these authentication failures every time someone uses ssh or scp! There were so many entries, we had to modify our configuration to ignore these messages. Now if there are real authentication failures we won't see them! This seems to be a case where the "cure" is worse than the disease! If the only solution to the information leak is to have this delay, then so be it. But it seems like this should definitely be a configuration option for those of us who aren't worried about this particular attack (the delay is very annoying). But the bogus authentication failure message is wrong in either case. As others have said, the cure is definitely worse than the disease. *** Bug 101662 has been marked as a duplicate of this bug. *** I'm hitting the same bug (this is definitely a bug - such a log entry should not exist) and I'm wondering why this message appears into the logs before the I even get the password prompt? What does it try to authenticate? Bug: 102174 is a duplicate of this bug. What will happen to this? We have a centralized syslog and we are getting all these bogus messages and also the authentication is slow now (using password or keys, it doesn't matter). There are any plans to fix this, or this issue will be ignored? How dangerous is not to apply the patch? JV. I think the results of the patch are far worse than the risk/problem it was meant to resolve. The patch may technically be correct and do what its supposed to but the behaviour it produces is flat out wrong - a successful login should not generate a failed login error message. No response from Red Hat since 8/5/2003 - that is LAME and I'm tired of having to back out patches that introduce worse problems than the ones they're intended to resolve (ala the OpenSSL errata blinding patch that breaks ubsec hardware crypto acceleration - see https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=88018 - its still marked as NEW nearly five months after being opened!!!!!!) if you add 'nodelay' to the pam auth for /etc/pam.d/sshd then the delay goes away entirely. not the best solution of course... Why is this marked closed? Has the bogus log entry been fixed? Has the delay been removed? I don't understand the point about this being intentional. The problem was that there was no delay when checking a nonexistant account. Surely the fix would be to make that go through PAM as well as successful logins. Successful logins should not be affected in any way. The openssh errata of openssh-3.5p1-9 (RHSA-2003:279-01) appears to fix the orginal two issues as well. Well done guys. This bug also afflicts RedHat Advanced Server 2.1. I couldn't possibly agree more with rocombs.edu: why is this marked closed? Has the delay been removed? Unfortunately, the answer is NO. And now that there's another OpenSSH bugfix in the pipeline, the worst-case scenario has arrived: sites like ours are faced with a choice of installing the broken, dog-slow version of OpenSSH RedHat is now distributing, or going without an important OpenSSH bug fix. I've filed a case against RHAS for this same bug, and I've been told that the bug--which they seemed to recognize as a bug, unlike in this call--must happen "upstream" (in the OpenSSH code, presumably). And that's it; no apparent further action. This is unacceptable, and RedHat needs to address this problem ASAP, especially with this important OpenSSH bugfix waiting in the pipeline. I should have added that the latest errata RPMs do not fix the slowdown bug (so I'm not sure what thm was talking about...). Both problems appear to be fixed for me.... (using RH9 with openssh-3.5p1-11). Maybe they fixed it in RHL but not in RHAS? What was your testcase? The new errata packages have fixed both problems on my Red Hat 7.1, 7.2 and 8.0 machines. Password-based (interactive) or public-key (passwordless) authentication. I'm testing between three verions: openssh-3.1p1-6 (which does not exhibit the bug), openssh-3.1p1-8 (where the bug first appeared), and openssl-3.1p1-14 (the current release level as of RHSA-2003:280, which still exhibits the bug). Simply by swapping RPMs, OpenSSH connections go from taking .4 seconds to 2.8 seconds. The difference seems to be that in RH9, the OpenSSH version is 3.5p1--so from what you're saying it sounds as though they've fixed this bug in that release. We're not that lucky in RHAS. But if they can fix it for that release, presumably the same fix could be backported to RHAS (just as the bug was backported). And to other 3.1p1-based releases (through RH7.3) as well. In fact, it would be interesting to know if anyone other than RH9 users have seen this issue resolved by the latest RPMs. It would seem that all 3.1p1- based releases should still be having the problem. I've just downloaded the OpenSSH 3.6.1p2-4 SRPM, dropped the 3.7.1p1 tarball in it and rebuilt. It was a clean swap with no other changes required. The delay has gone...and problem fixed. Lets hope that 3.7.1p1 appers in Rawhide real soon.. It looks like the new errata rpms have fixed both problems, but only if you are using SSH2 to connect. Connections using SSH1 are still subject to the delay and authentication failure message in the logs. Philip's comment applies to the RHAS RPMs as well: SSH1 still exhibits both bugs, but SSH2 works normally again. And in fact the SRPM files for 7.2 and RHAS are actually the same file, so it wouldn't make sense for them to behave differently. It would be interesting to know if RH9 (with openssh 3.5p1) exhibits the same behavior. I see the following in the changelog from the openssh.spec file: * Tue Sep 16 2003 Nalin Dahyabhai <nalin> 3.1p1-9 - apply patch to store the correct buffer size in allocated buffers (CAN-2003-0693) - skip the initial PAM authentication attempt with an empty password if empty passwords are not permitted in our configuration (#103998) So it looks like Nalin did throw in a fix for the bugs, but didn't update this particular bugID (not a surprise since it's "CLOSED"). But, his fix must not have addressed SSH1. So Nalin, if you're watching this closed but very active bug: thanks for the SSH2 fix, and could you please see about making your bugfix work with SSH1 as well? In my RHAS suport case, I finally persuaded RedHat to fix this issue for SSH1
as well as SSH2. Unfortunately, they say that they will NOT be rereleasing the
RPMs for RHAS or most other RedHat trains, so to get the SSH1 fix you'll need
to rebuild from the SRPM using an updated patch file (there are two, one for
3.5p1 and one for 3.1p1; I'll attach them to this case). Nalin claims that
you'll see this in RawHide even before a new OpenSSH RPM is released, but
apparently all other releases will have to wait until there's yet another bug
in OpenSSH before they get this fix as part of an official RedHat RPM.
I'm glad that this has now been addressed, but I find it a bit irritating that
1) this case never had its NOTABUG status changed even though RedHat/Nalin
clearly do recognize that it was a bug (as of course it was), 2) RedHat is
trying to lay low with this bug and fix by not re-releasing the RPMs now that
SSH1 has been repaired, which causes more work for all of us to get in put in
place, and 3) I've had to go through this effort to get the patch files
published in a place where other people can use them when it's something Nalin
should be doing. Maybe I should send RedHat an invoice at my normal hourly
rate?
Below is Nalin's email to the person who was handling my RHAS case, with
instructions on how to rebuild.
(Nalin, if you're watching this, please change the status to something more
appropriate so that people searching Bugzilla for this issue in the future will
be able to tell that it does in fact contain a fix for their problem.)
-----Forwarded Message-----
> From: Nalin Dahyabhai <nalin>
> To: Chris Kloiber <ckloiber>
> Subject: OpenSSH - updated skip-initial patch
> Date: Tue, 23 Sep 2003 16:34:56 -0400
>
> Chris, this patch adds the necessary hunk for
> making SSH1 skip the initial authentication
> attempt, as we did for SSH2 in the last erratum.
> It's checked in on the appropriate branch, so if
> another OpenSSH update comes down the
> pipeline, it'll be incorporated, but until then
> you'll only see this in Raw Hide.
>
> To use it, install the source package with 'rpm -
> U', replace the patch file in
> /src/redhat/SOURCES with the attached patch
> [1], and run 'rpmbuild -bb
> /usr/src/redhat/SPECS/openssh.spec'.
> Assuming no build failures due to unmet build
> dependencies or other unforeseen problems,
> you'll get a modified version of the package we
> last shipped out which has the desired change
> incorporated into it.
>
> HTH,
>
> Nalin
>
> [1] There are two, actually, because the changes > for 3.1p1 are slightly
> different from the
> changes for 3.4p1, 3.5p1, and 3.6.1p2.
----End Forwarded Message-----
Created attachment 94698 [details]
Updated openssh-3.1p1-skip-initial.patch file
Created attachment 94699 [details]
Updated openssh-3.5p1-skip-initial.patch file
*** Bug 103935 has been marked as a duplicate of this bug. *** *** Bug 102174 has been marked as a duplicate of this bug. *** *** Bug 101252 has been marked as a duplicate of this bug. *** |