Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1558977 - NFS mounts failing when keytab present
Summary: NFS mounts failing when keytab present
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 27
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-03-21 12:44 UTC by Michael Young
Modified: 2018-04-06 11:08 UTC (History)
18 users (show)

Fixed In Version: kernel-4.15.14-300.fc27 kernel-4.16.0-300.fc28
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-04-06 11:08:41 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Michael Young 2018-03-21 12:44:39 UTC
NFS mounts on of my computers with kerberos set up have stopped working after a kernel update. It can mount filesystems successfully when running 4.15.3-300.fc27.x86_64 but stops working with 4.15.4-300.fc27.x86_64 and later. It does work if I rename /etc/krb5.keytab to disable kerberos. This is true even when mounting with sec=sys where it shouldn't be trying to use kerberos at all (though my testing suggests it is trying to mount with kerberos even when sec=sys is requested). The error I get when trying a mount command is
mount.nfs: mount system call failed

Comment 1 Jeremy Cline 2018-03-21 13:40:42 UTC
Hi,

There are a couple NFS-related changes in 4.15.4 - I've reverted the one that looks the most likely to me. Can you try the kernel from https://koji.fedoraproject.org/koji/taskinfo?taskID=25858412 when it is finished building?

Is there anything in the logs in either the client or the server when you try to mount the filesystem? I assume the client is Fedora, what's the server? Finally, have you tested the latest 4.15 kernel from updates-testing? It would also be helpful to test the latest 4.16 kernel (https://bodhi.fedoraproject.org/updates/FEDORA-2018-5ffc6f479f) to see if it's already been fixed.

Thanks!

Comment 2 Michael Young 2018-03-21 13:55:02 UTC
There are errors in the client such as the following in dmesg
[ 1232.522816] gss_marshal: gss_get_mic FAILED (851968)
[ 1232.522819] RPC: couldn't encode RPC header, exit EIO
[ 1232.522856] gss_marshal: gss_get_mic FAILED (851968)
[ 1232.522857] RPC: couldn't encode RPC header, exit EIO
[ 1232.522863] NFS: nfs4_discover_server_trunking unhandled error -5. Exiting with error EIO
[ 1232.525039] gss_marshal: gss_get_mic FAILED (851968)
[ 1232.525042] RPC: couldn't encode RPC header, exit EIO
There aren't any errors on the server (but it doesn't log much as it is a busy server).

The issue is also present in 4.16 - I have tested kernel-4.16.0-0.rc1.git0.1.fc28.x86_64 and kernel-4.16.0-0.rc6.git0.2.fc28.x86_64 while investigating if it was fixed in Fedora 28.

Comment 3 Michael Young 2018-03-21 16:38:31 UTC
4.15.4-301.fc27.x86_64 still shows the problem.

Comment 4 Jeremy Cline 2018-03-21 23:56:53 UTC
Thanks for testing that.

Nothing else looks very suspicious to me. Since you have a straight-forward reproducer and a known good version, I recommend bisecting the upstream kernel to find the commit that introduced the problem.

Comment 5 Michael Young 2018-03-25 00:12:26 UTC
I traced the problem patch which is
[46e8d06e423c4f35eac7a8b677b713b3ec9b0684] crypto: hash - prevent using keyed hashes without setting key
NFS works for me with kernel-4.15.12-301.fc27.x86_64 built with this patch reverted.

Comment 6 Jeremy Cline 2018-03-26 18:43:35 UTC
Great!

The next thing to do is report this to upstream. I'm happy to do this if you don't want to, but it's simpler if you do it since I don't have a system to reproduce this on at the moment.

I recommend sending an email to the original committer and reviewer to the patch that introduced the problem and CC the maintainers of NFS and the relevant lists:

Author and reviewer:
* Eric Biggers <ebiggers>
* Herbert Xu <herbert.org.au>

NFS maintainers:

* "J. Bruce Fields" <bfields>
* Jeff Layton <jlayton>
* Trond Myklebust <trond.myklebust>
* Anna Schumaker <anna.schumaker>

Lists:
* linux-nfs.org
* netdev.org
* linux-kernel.org

Comment 7 Michael Young 2018-03-28 08:09:45 UTC
The patch posted upstream for testing in
https://marc.info/?l=linux-kernel&m=152218980114708&w=2
fixes the problem.

Comment 8 Jeremy Cline 2018-03-28 13:41:22 UTC
Great! When it's accepted upstream I'll add it to the stable builds. The updates system will post on this bug when it's in the updates-testing repository.

Comment 9 Fedora Update System 2018-03-30 01:38:17 UTC
kernel-4.15.14-200.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2018-18754260e4

Comment 10 Fedora Update System 2018-03-30 01:39:43 UTC
kernel-4.15.14-300.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-7802740586

Comment 11 Fedora Update System 2018-03-30 14:38:56 UTC
kernel-4.15.14-200.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-18754260e4

Comment 12 Fedora Update System 2018-03-30 14:48:27 UTC
kernel-4.15.14-300.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-7802740586

Comment 13 Fedora Update System 2018-04-02 19:16:18 UTC
kernel-4.16.0-1.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-cf40575a4e

Comment 14 Fedora Update System 2018-04-03 14:24:54 UTC
kernel-4.15.14-200.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

Comment 15 Fedora Update System 2018-04-03 14:51:56 UTC
kernel-4.15.14-300.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

Comment 16 Fedora Update System 2018-04-03 15:57:58 UTC
kernel-4.16.0-300.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-cf40575a4e

Comment 17 Fedora Update System 2018-04-06 11:08:41 UTC
kernel-4.16.0-300.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.