Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1363924
Summary: | getgrouplist() returns data not backed by the initgroups record of nsswitch.conf if nscd is running | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | Scott Poore <spoore> |
Component: | glibc | Assignee: | glibc team <glibc-bugzilla> |
Status: | CLOSED UPSTREAM | QA Contact: | qe-baseos-tools-bugs |
Severity: | unspecified | Docs Contact: | |
Priority: | unspecified | ||
Version: | 7.3 | CC: | ashankar, codonell, fweimer, kdudka, mnewsome, pfrankli, spoore |
Target Milestone: | rc | ||
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2020-04-16 12:44:28 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Attachments: |
Description
Scott Poore
2016-08-04 00:44:19 UTC
I'm marking this one as Regression because this was not the behavior from RHEL7.2. [root@rhel7-2 yum.repos.d]# rpm -q coreutils glibc coreutils-8.22-15.el7.x86_64 glibc-2.17-105.el7.x86_64 [root@rhel7-2 yum.repos.d]# ps -ef|grep nscd root 14569 2416 0 20:01 pts/0 00:00:00 grep --color=auto nscd [root@rhel7-2 yum.repos.d]# id ypuser1 uid=973200006(ypuser1) gid=973200006(ypuser1) groups=973200006(ypuser1),973200007(ypgroup1) [root@rhel7-2 yum.repos.d]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.2 (Maipo) The above is a RHEL7.2 host connected to the same NIS enabled IPA server as a separate 7.3 host that showed the failure. What is the output of 'getent group ypgroup1' before/after starting nscd? Kamil, It appears that getent shows the same before and after starting nscd: [root@rhel7-3 ~]# id ypuser1 uid=973200006(ypuser1) gid=973200006(ypuser1) groups=973200006(ypuser1) [root@rhel7-3 ~]# getent group ypgroup1 ypgroup1:*:973200007:ypuser1 [root@rhel7-3 ~]# systemctl start nscd [root@rhel7-3 ~]# id ypuser1 uid=973200006(ypuser1) gid=973200006(ypuser1) groups=973200007(ypgroup1),973200006(ypuser1) [root@rhel7-3 ~]# getent group ypgroup1 ypgroup1:*:973200007:ypuser1 I tried the Steps to Reproduce from comment #0 (Thank you for making them explicit!) on a single machine but it did not reproduce the problem on a fresh el7 host with glibc-2.17-156.el7.x86_64 and coreutils-8.22-18.el7.x86_64. Is it necessary to use separate machines for server and client to reproduce it? Could you please attach the output of 'rpm -a | sort -V' from the client? Created attachment 1187863 [details]
list of packages installed when _unsuccesfully_ tried to reproduce the bug
Created attachment 1187916 [details]
list of packages installed on separate client when seeing the bug
Yes, it is necessary I think to run separately. The IPA server installs an IPA client by default which configures SSSD. So, unless something else is happening, I suspect that SSSD may be doing the same thing nscd is in my case. You might be able to reproduce it on a single host by also stopping sssd service but, I haven't tried that yet. In my case though, I was seeing it on a separate host. Thanks, Scott (In reply to Scott Poore from comment #9) > Created attachment 1187916 [details] > list of packages installed on separate client when seeing the bug I see you have installed some *beta1* packages, which arises suspicion: $ curl -s https://bugzilla.redhat.com/attachment.cgi?id=1187916 | grep beta libipa_hbac-1.14.0-0.2.beta1.el7.x86_64 libsss_autofs-1.14.0-0.2.beta1.el7.x86_64 libsss_idmap-1.14.0-0.2.beta1.el7.x86_64 libsss_nss_idmap-1.14.0-0.2.beta1.el7.x86_64 python-libipa_hbac-1.14.0-0.2.beta1.el7.x86_64 python-sssdconfig-1.14.0-0.2.beta1.el7.noarch python-sss-murmur-1.14.0-0.2.beta1.el7.x86_64 sssd-1.14.0-0.2.beta1.el7.x86_64 sssd-ad-1.14.0-0.2.beta1.el7.x86_64 sssd-client-1.14.0-0.2.beta1.el7.x86_64 sssd-common-1.14.0-0.2.beta1.el7.x86_64 sssd-common-pac-1.14.0-0.2.beta1.el7.x86_64 sssd-ipa-1.14.0-0.2.beta1.el7.x86_64 sssd-krb5-1.14.0-0.2.beta1.el7.x86_64 sssd-krb5-common-1.14.0-0.2.beta1.el7.x86_64 sssd-ldap-1.14.0-0.2.beta1.el7.x86_64 sssd-proxy-1.14.0-0.2.beta1.el7.x86_64 Could you please try to update them to the packages from sssd-1.14.0-14.el7? Created attachment 1187937 [details]
upgraded list of packages after installed on separate client when seeing the bug
yeah, I noticed the beta packages too when I compared the lists and started an upgrade. here is the result.
Created attachment 1187940 [details]
strace with more info for id command without nscd running fails
Created attachment 1187941 [details]
strace with more info for id command without nscd running on rhel7.2 host works
Something interesting I just found comparing strace output from the host failing and a RHEL7.2 host where it works: It appears that where it's failing, I do not see a NIS lookup for group.byname. So it never pulls the full list of NIS groups and finds the user in another group. Does that help? (In reply to Scott Poore from comment #0) > 2. Setup NIS client on another machine > > yum -y install ypbind yp-tools rpcbind nscd > systemctl stop firewalld > IPA_SERVER_HOSTNAME=<enter hostname of above server> > echo "domain example.com server $IPA_SERVER_HOSTNAME" >> /etc/yp.conf I needed to replace example.com by nis.test for ypbind to start when server and client were two different machines. Is the change correct? Anyway, I was not able to reproduce it even with client on a separate machine. The first invocation of 'id ypuser1' already provides the expected result: # systemctl stop firewalld # echo "domain nis.test server $IPA_SERVER_HOSTNAME" >> /etc/yp.conf # echo "NISDOMAIN=nis.test" >> /etc/sysconfig/network # sed -i 's/^passwd:.*$/passwd: files nis/' /etc/nsswitch.conf # sed -i 's/^shadow:.*$/shadow: files nis/' /etc/nsswitch.conf # sed -i 's/^group:.*$/group: files nis/' /etc/nsswitch.conf # nisdomainname nis.test # systemctl restart rpcbind # systemctl start ypbind # id ypuser1 uid=817000001(ypuser1) gid=817000001(ypuser1) groups=817000001(ypuser1),817000003(ypgroup1) nscd was not running and sssd was not installed on the client. yes, that's correct. Sorry about the inconsistency there. Where I saw that issue was on a pretty basic VM. I'm setting up another environment to see but, I expect I'll see it there too as this was originally found from automated tests. What else may help troubleshoot this? I'm not quite sure what is different about your setup but, I'm able to reproduce this in a different environment. ######### On Master: [root@qe-blade-07 ~]# ipa-server-install -n nis.test -r NIS.TEST -a Secret123 -p Secret123 -U ...truncated output... [root@qe-blade-07 ~]# echo Secret123 | ipa-compat-manage enable Directory Manager password: Plugin already Enabled [root@qe-blade-07 ~]# echo Secret123 | ipa-nis-manage enable Directory Manager password: Enabling plugin This setting will not take effect until you restart Directory Server. The rpcbind service may need to be started. [root@qe-blade-07 ~]# systemctl restart dirsrv.target [root@qe-blade-07 ~]# systemctl restart rpcbind [root@qe-blade-07 ~]# kinit admin Password for admin: [root@qe-blade-07 ~]# ipa user-add ypuser1 --first=f --last=l -------------------- Added user "ypuser1" -------------------- User login: ypuser1 First name: f Last name: l Full name: f l Display name: f l Initials: fl Home directory: /home/ypuser1 GECOS: f l Login shell: /bin/sh Principal name: ypuser1 Principal alias: ypuser1 Email address: ypuser1 UID: 964200001 GID: 964200001 Password: False Member of groups: ipausers Kerberos keys available: False [root@qe-blade-07 ~]# ipa group-add ypgroup1 ---------------------- Added group "ypgroup1" ---------------------- Group name: ypgroup1 GID: 964200003 [root@qe-blade-07 ~]# ipa group-add-member ypgroup1 --users=ypuser1 Group name: ypgroup1 GID: 964200003 Member users: ypuser1 ------------------------- Number of members added 1 ------------------------- ############ On NIS Client [root@qe-blade-08 ~]# yum -y install ypbind yp-tools rpcbind nscd Loaded plugins: product-id, search-disabled-repos, subscription-manager This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. Package rpcbind-0.2.0-36.el7.x86_64 already installed and latest version Resolving Dependencies --> Running transaction check ---> Package nscd.x86_64 0:2.17-156.el7 will be installed ---> Package yp-tools.x86_64 0:2.14-3.el7 will be installed ---> Package ypbind.x86_64 3:1.37.1-7.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ======================================================================================================= Package Arch Version Repository Size ======================================================================================================= Installing: nscd x86_64 2.17-156.el7 beaker-Server 267 k yp-tools x86_64 2.14-3.el7 beaker-Server 79 k ypbind x86_64 3:1.37.1-7.el7 beaker-Server 62 k Transaction Summary ======================================================================================================= Install 3 Packages Total download size: 408 k Installed size: 478 k Downloading packages: (1/3): yp-tools-2.14-3.el7.x86_64.rpm | 79 kB 00:00:00 (2/3): nscd-2.17-156.el7.x86_64.rpm | 267 kB 00:00:00 (3/3): ypbind-1.37.1-7.el7.x86_64.rpm | 62 kB 00:00:00 ------------------------------------------------------------------------------------------------------- Total 3.4 MB/s | 408 kB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : 3:ypbind-1.37.1-7.el7.x86_64 1/3 Installing : yp-tools-2.14-3.el7.x86_64 2/3 Installing : nscd-2.17-156.el7.x86_64 3/3 Verifying : nscd-2.17-156.el7.x86_64 1/3 Verifying : yp-tools-2.14-3.el7.x86_64 2/3 Verifying : 3:ypbind-1.37.1-7.el7.x86_64 3/3 Installed: nscd.x86_64 0:2.17-156.el7 yp-tools.x86_64 0:2.14-3.el7 ypbind.x86_64 3:1.37.1-7.el7 Complete! [root@qe-blade-08 ~]# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@qe-blade-08 ~]# echo "domain nis.test server $IPA_SERVER_HOSTNAME" >> /etc/yp.conf [root@qe-blade-08 ~]# vi /etc/yp.conf [root@qe-blade-08 ~]# export IPA_SERVER_HOSTNAME=<entered hostname here> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^note the above line was edited^^^^^^^^^^^^^^^^ [root@qe-blade-08 ~]# echo "domain nis.test server $IPA_SERVER_HOSTNAME" >> /etc/yp.conf [root@qe-blade-08 ~]# echo "NISDOMAIN=nis.test" >> /etc/sysconfig/network [root@qe-blade-08 ~]# sed -i 's/^passwd:.*$/passwd: files nis/' /etc/nsswitch.conf [root@qe-blade-08 ~]# sed -i 's/^shadow:.*$/shadow: files nis/' /etc/nsswitch.conf [root@qe-blade-08 ~]# sed -i 's/^group:.*$/group: files nis/' /etc/nsswitch.conf [root@qe-blade-08 ~]# nisdomainname nis.test [root@qe-blade-08 ~]# systemctl restart rpcbind [root@qe-blade-08 ~]# systemctl start ypbind [root@qe-blade-08 ~]# ps -ef|grep nscd root 29031 28839 0 21:07 pts/0 00:00:00 grep --color=auto nscd [root@qe-blade-08 ~]# ps -ef|grep sssd root 29033 28839 0 21:07 pts/0 00:00:00 grep --color=auto sssd [root@qe-blade-08 ~]# id ypuser1 uid=964200001(ypuser1) gid=964200001(ypuser1) groups=964200001(ypuser1) [root@qe-blade-08 ~]# ypcat group|grep ypuser1 ypgroup1:*:964200003:ypuser1 ypuser1:*:964200001: [root@qe-blade-08 ~]# getent group ypgroup1 ypgroup1:*:964200003:ypuser1 [root@qe-blade-08 ~]# id ypuser1 uid=964200001(ypuser1) gid=964200001(ypuser1) groups=964200001(ypuser1) [root@qe-blade-08 ~]# ypcat group|grep ypuser1 ypgroup1:*:964200003:ypuser1 ypuser1:*:964200001: [root@qe-blade-08 ~]# getent group ypgroup1 ypgroup1:*:964200003:ypuser1 Kamil, I can provide you with hosts where the problem occurs if that helps. Let me know. Thanks, Scott (In reply to Scott Poore from comment #18) > I'm not quite sure what is different about your setup but, I'm able to > reproduce this in a different environment. I tried exactly the steps from comment #18. The output was equivalent up to the first invocation of 'id ypuser1' where it already provided the expected result: # id ypuser1 uid=1331200001(ypuser1) gid=1331200001(ypuser1) groups=1331200001(ypuser1),1331200003(ypgroup1) (In reply to Scott Poore from comment #19) > I can provide you with hosts where the problem occurs if that helps. That would be really appreciated! I have verified that id(1) correctly prints what getgrouplist(3) returns. It returns nothing but the group argument it got on its output. This seems to be inconsistent with the output of getent(1), which says that 'ypuser1' is member of 'ypgroup1': # getent group ypgroup1 ypgroup1:*:745800003:ypuser1 strace shows that /lib64/libnss_nis.so.2 is not opened upon invocation of getgrouplist(3) despite it 'nis' is enabled in /etc/nsswitch.conf for group lookup: # grep ^group /etc/nsswitch.conf group: files nis # strace -e trace=open ./a.out open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 open("/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 3 open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 open("/lib64/libnss_files.so.2", O_RDONLY|O_CLOEXEC) = 3 open("/etc/group", O_RDONLY|O_CLOEXEC) = 3 666 +++ exited with 0 +++ On the other hand, getent opens /lib64/libnss_nis.so.2: # strace -e trace=open getent group ypgroup1 open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 open("/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 3 open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 open("/lib64/libnss_files.so.2", O_RDONLY|O_CLOEXEC) = 3 open("/etc/group", O_RDONLY|O_CLOEXEC) = 3 open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 open("/lib64/libnss_nis.so.2", O_RDONLY|O_CLOEXEC) = 3 open("/lib64/libnsl.so.1", O_RDONLY|O_CLOEXEC) = 3 open("/var/yp/binding/nis.test.2", O_RDONLY) = 3 ypgroup1:*:745800003:ypuser1 +++ exited with 0 +++ I am switching the component to glibc... Created attachment 1190824 [details]
minimal example (lookup only)
# curl -JO 'https://bugzilla.redhat.com/attachment.cgi?id=1190824' # sh list-groups.c + getent group ypgroup1 ypgroup1:*:745800003:ypuser1 + gcc -Wall -Wextra list-groups.c + ./a.out 666 + exit 0 # service nscd start Redirecting to /bin/systemctl start nscd.service # sh list-groups.c + getent group ypgroup1 ypgroup1:*:745800003:ypuser1 + gcc -Wall -Wextra list-groups.c + ./a.out 745800003 666 + exit 0 # rpm -q glibc glibc-2.17-156.el7.x86_64 Note that the issue recurs if nscd is stopped again: # service nscd stop Redirecting to /bin/systemctl stop nscd.service # sh list-groups.c + getent group ypgroup1 ypgroup1:*:745800003:ypuser1 + gcc -Wall -Wextra list-groups.c + ./a.out 666 + exit 0 From the /etc/nsswitch.conf file of the test client system: initgroups: files You need add “nis” there as well, not just on the “group” line. Arguably, it is an nscd bug that you see NIS data with getgrouplist under such circumstances. (In reply to Florian Weimer from comment #27) > From the /etc/nsswitch.conf file of the test client system: > > initgroups: files > > You need add “nis” there as well, not just on the “group” line. Indeed. It is documented in the nsswitch.conf(5) man page. My mistake. > Arguably, it is an nscd bug that you see NIS data with getgrouplist under > such circumstances. That is exactly what made it less obvious I guess... Yes, I do see that adding nis to initgroups fixes the issue. Also, commenting it out altogether resolves the issue as well: # grep initgroups /etc/nsswitch.conf initgroups: files # id ypuser1 uid=745800001(ypuser1) gid=745800001(ypuser1) groups=745800001(ypuser1) # sed -i 's/^.*\(initgroups\)/#\1/' /etc/nsswitch.conf ; grep initgroups /etc/nsswitch.conf #initgroups: files # id ypuser1 uid=745800001(ypuser1) gid=745800001(ypuser1) groups=745800001(ypuser1),745800003(ypgroup1) I went back and checked again on a fresh, minimal RHEL 7.2 install and that had initgroups commented out: # rpm -q glibc glibc-2.17-105.el7.x86_64 # grep initgroups /etc/nsswitch.conf #initgroups: files So, was there a change in glibc to uncomment that in the latest release? Thanks, Scott (In reply to Scott Poore from comment #31) > So, was there a change in glibc to uncomment that in the latest release? The file has not been changed within Red Hat Enterprise Linux 7 on the glibc side. I have older installations which contain the line “initgroups: files”, too. Please note that bug 1294574 is quite different from this bug; it is specific to Fedora. It was not present in glibc 2.17. I think I may have more information about my issue. My test was running the following: authconfig --enablekrb5 --update That is before the id command did not return all expected groups. On a RHEL 7.2 server I did see that initgroups was not commented on clean build. But, I installed IPA and then it was commented (by ipa-server-install). I uncommented it and then just ran that authconfig command and it was commented again: [root@master ~]# grep initgroups /etc/nsswitch.conf #initgroups: files [root@master ~]# vi /etc/nsswitch.conf [root@master ~]# grep initgroups /etc/nsswitch.conf initgroups: files [root@master ~]# authconfig --enablekrb5 --update [root@master ~]# grep initgroups /etc/nsswitch.conf #initgroups: files In at least some of my tests I noticed later that authconfig was missing. I modified my tests to specifically install it in case a dependency changed. I've removed another workaround that runs nscd and I'm running a test to see if everything is clear now from my tests. This turns out to be a bug in the nscd configuration, we need to employ the same fallback logic as is used in the non-nscd code. Currently nscd just uses groups data with no initgroups. We are tracking this bug upstream in this issue: https://sourceware.org/bugzilla/show_bug.cgi?id=25835 We will be working on the bug upstream, fixing it there, and once the fix is available we can backport to RHEL7 or RHEL8 depending on requirements. I'm marking this CLOSED/UPSTREAM. |