I'm marking this one as Regression because this was not the behavior from RHEL7.2.

[root@rhel7-2 yum.repos.d]# rpm -q coreutils glibc
coreutils-8.22-15.el7.x86_64
glibc-2.17-105.el7.x86_64

[root@rhel7-2 yum.repos.d]# ps -ef|grep nscd
root     14569  2416  0 20:01 pts/0    00:00:00 grep --color=auto nscd

[root@rhel7-2 yum.repos.d]# id ypuser1
uid=973200006(ypuser1) gid=973200006(ypuser1) groups=973200006(ypuser1),973200007(ypgroup1)

[root@rhel7-2 yum.repos.d]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.2 (Maipo)

The above is a RHEL7.2 host connected to the same NIS enabled IPA server as a separate 7.3 host that showed the failure.

What is the output of 'getent group ypgroup1' before/after starting nscd?

Kamil,

It appears that getent shows the same before and after starting nscd:

[root@rhel7-3 ~]# id ypuser1
uid=973200006(ypuser1) gid=973200006(ypuser1) groups=973200006(ypuser1)

[root@rhel7-3 ~]# getent group ypgroup1
ypgroup1:*:973200007:ypuser1

[root@rhel7-3 ~]# systemctl start nscd

[root@rhel7-3 ~]# id ypuser1
uid=973200006(ypuser1) gid=973200006(ypuser1) groups=973200007(ypgroup1),973200006(ypuser1)

[root@rhel7-3 ~]# getent group ypgroup1
ypgroup1:*:973200007:ypuser1

I tried the Steps to Reproduce from comment #0 (Thank you for making them explicit!) on a single machine but it did not reproduce the problem on a fresh el7 host with glibc-2.17-156.el7.x86_64 and coreutils-8.22-18.el7.x86_64.

Is it necessary to use separate machines for server and client to reproduce it?

Could you please attach the output of 'rpm -a | sort -V' from the client?

Created attachment 1187863 [details]
list of packages installed when _unsuccesfully_ tried to reproduce the bug

Created attachment 1187916 [details]
list of packages installed on separate client when seeing the bug

Yes, it is necessary I think to run separately.  The IPA server installs an IPA client by default which configures SSSD.  So, unless something else is happening, I suspect that SSSD may be doing the same thing nscd is in my case.  You might be able to reproduce it on a single host by also stopping sssd service but, I haven't tried that yet.  In my case though, I was seeing it on a separate host.

Thanks,
Scott

(In reply to Scott Poore from comment #9)
> Created attachment 1187916 [details]
> list of packages installed on separate client when seeing the bug

I see you have installed some *beta1* packages, which arises suspicion:

$ curl -s https://bugzilla.redhat.com/attachment.cgi?id=1187916 | grep beta
libipa_hbac-1.14.0-0.2.beta1.el7.x86_64
libsss_autofs-1.14.0-0.2.beta1.el7.x86_64
libsss_idmap-1.14.0-0.2.beta1.el7.x86_64
libsss_nss_idmap-1.14.0-0.2.beta1.el7.x86_64
python-libipa_hbac-1.14.0-0.2.beta1.el7.x86_64
python-sssdconfig-1.14.0-0.2.beta1.el7.noarch
python-sss-murmur-1.14.0-0.2.beta1.el7.x86_64
sssd-1.14.0-0.2.beta1.el7.x86_64
sssd-ad-1.14.0-0.2.beta1.el7.x86_64
sssd-client-1.14.0-0.2.beta1.el7.x86_64
sssd-common-1.14.0-0.2.beta1.el7.x86_64
sssd-common-pac-1.14.0-0.2.beta1.el7.x86_64
sssd-ipa-1.14.0-0.2.beta1.el7.x86_64
sssd-krb5-1.14.0-0.2.beta1.el7.x86_64
sssd-krb5-common-1.14.0-0.2.beta1.el7.x86_64
sssd-ldap-1.14.0-0.2.beta1.el7.x86_64
sssd-proxy-1.14.0-0.2.beta1.el7.x86_64

Could you please try to update them to the packages from sssd-1.14.0-14.el7?

Created attachment 1187937 [details]
upgraded list of packages after installed on separate client when seeing the bug

yeah, I noticed the beta packages too when I compared the lists and started an upgrade.  here is the result.

Created attachment 1187940 [details]
strace with more info for id command without nscd running fails

Created attachment 1187941 [details]
strace with more info for id command without nscd running on rhel7.2 host works

Something interesting I just found comparing strace output from the host failing and a RHEL7.2 host where it works:

It appears that where it's failing, I do not see a NIS lookup for group.byname.  So it never pulls the full list of NIS groups and finds the user in another group.

Does that help?

(In reply to Scott Poore from comment #0)
> 2.  Setup NIS client on another machine
> 
> yum -y install ypbind yp-tools rpcbind nscd
> systemctl stop firewalld
> IPA_SERVER_HOSTNAME=<enter hostname of above server>
> echo "domain example.com server $IPA_SERVER_HOSTNAME" >> /etc/yp.conf

I needed to replace example.com by nis.test for ypbind to start when server and client were two different machines.  Is the change correct?

Anyway, I was not able to reproduce it even with client on a separate machine.  The first invocation of 'id ypuser1' already provides the expected result:

# systemctl stop firewalld
# echo "domain nis.test server $IPA_SERVER_HOSTNAME" >> /etc/yp.conf
# echo "NISDOMAIN=nis.test" >> /etc/sysconfig/network
# sed -i 's/^passwd:.*$/passwd: files nis/' /etc/nsswitch.conf 
# sed -i 's/^shadow:.*$/shadow: files nis/' /etc/nsswitch.conf 
# sed -i 's/^group:.*$/group:  files nis/' /etc/nsswitch.conf 
# nisdomainname nis.test
# systemctl restart rpcbind
# systemctl start ypbind
# id ypuser1
uid=817000001(ypuser1) gid=817000001(ypuser1) groups=817000001(ypuser1),817000003(ypgroup1)

nscd was not running and sssd was not installed on the client.

yes, that's correct.   Sorry about the inconsistency there.  Where I saw that issue was on a pretty basic VM.  I'm setting up another environment to see but, I expect I'll see it there too as this was originally found from automated tests.

What else may help troubleshoot this?

I'm not quite sure what is different about your setup but, I'm able to reproduce this in a different environment.


######### On Master:

[root@qe-blade-07 ~]# ipa-server-install -n nis.test -r NIS.TEST -a Secret123 -p Secret123 -U
...truncated output...

[root@qe-blade-07 ~]# echo Secret123 | ipa-compat-manage enable
Directory Manager password: 
Plugin already Enabled

[root@qe-blade-07 ~]# echo Secret123 | ipa-nis-manage enable
Directory Manager password: 
Enabling plugin
This setting will not take effect until you restart Directory Server.
The rpcbind service may need to be started.

[root@qe-blade-07 ~]# systemctl restart dirsrv.target

[root@qe-blade-07 ~]# systemctl restart rpcbind

[root@qe-blade-07 ~]# kinit admin
Password for admin: 

[root@qe-blade-07 ~]# ipa user-add ypuser1 --first=f --last=l
--------------------
Added user "ypuser1"
--------------------
  User login: ypuser1
  First name: f
  Last name: l
  Full name: f l
  Display name: f l
  Initials: fl
  Home directory: /home/ypuser1
  GECOS: f l
  Login shell: /bin/sh
  Principal name: ypuser1
  Principal alias: ypuser1
  Email address: ypuser1
  UID: 964200001
  GID: 964200001
  Password: False
  Member of groups: ipausers
  Kerberos keys available: False

[root@qe-blade-07 ~]# ipa group-add ypgroup1
----------------------
Added group "ypgroup1"
----------------------
  Group name: ypgroup1
  GID: 964200003
[root@qe-blade-07 ~]# ipa group-add-member ypgroup1 --users=ypuser1
  Group name: ypgroup1
  GID: 964200003
  Member users: ypuser1
-------------------------
Number of members added 1
-------------------------


############ On NIS Client

[root@qe-blade-08 ~]# yum -y install ypbind yp-tools rpcbind nscd
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Package rpcbind-0.2.0-36.el7.x86_64 already installed and latest version
Resolving Dependencies
--> Running transaction check
---> Package nscd.x86_64 0:2.17-156.el7 will be installed
---> Package yp-tools.x86_64 0:2.14-3.el7 will be installed
---> Package ypbind.x86_64 3:1.37.1-7.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=======================================================================================================
 Package              Arch               Version                       Repository                 Size
=======================================================================================================
Installing:
 nscd                 x86_64             2.17-156.el7                  beaker-Server             267 k
 yp-tools             x86_64             2.14-3.el7                    beaker-Server              79 k
 ypbind               x86_64             3:1.37.1-7.el7                beaker-Server              62 k

Transaction Summary
=======================================================================================================
Install  3 Packages

Total download size: 408 k
Installed size: 478 k
Downloading packages:
(1/3): yp-tools-2.14-3.el7.x86_64.rpm                                           |  79 kB  00:00:00     
(2/3): nscd-2.17-156.el7.x86_64.rpm                                             | 267 kB  00:00:00     
(3/3): ypbind-1.37.1-7.el7.x86_64.rpm                                           |  62 kB  00:00:00     
-------------------------------------------------------------------------------------------------------
Total                                                                  3.4 MB/s | 408 kB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : 3:ypbind-1.37.1-7.el7.x86_64                                                        1/3 
  Installing : yp-tools-2.14-3.el7.x86_64                                                          2/3 
  Installing : nscd-2.17-156.el7.x86_64                                                            3/3 
  Verifying  : nscd-2.17-156.el7.x86_64                                                            1/3 
  Verifying  : yp-tools-2.14-3.el7.x86_64                                                          2/3 
  Verifying  : 3:ypbind-1.37.1-7.el7.x86_64                                                        3/3 

Installed:
  nscd.x86_64 0:2.17-156.el7      yp-tools.x86_64 0:2.14-3.el7      ypbind.x86_64 3:1.37.1-7.el7     

Complete!

[root@qe-blade-08 ~]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

[root@qe-blade-08 ~]# echo "domain nis.test server $IPA_SERVER_HOSTNAME" >> /etc/yp.conf

[root@qe-blade-08 ~]# vi /etc/yp.conf 

[root@qe-blade-08 ~]# export IPA_SERVER_HOSTNAME=<entered hostname here>

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^note the above line was edited^^^^^^^^^^^^^^^^

[root@qe-blade-08 ~]# echo "domain nis.test server $IPA_SERVER_HOSTNAME" >> /etc/yp.conf

[root@qe-blade-08 ~]# echo "NISDOMAIN=nis.test" >> /etc/sysconfig/network

[root@qe-blade-08 ~]# sed -i 's/^passwd:.*$/passwd: files nis/' /etc/nsswitch.conf

[root@qe-blade-08 ~]# sed -i 's/^shadow:.*$/shadow: files nis/' /etc/nsswitch.conf

[root@qe-blade-08 ~]# sed -i 's/^group:.*$/group:  files nis/' /etc/nsswitch.conf

[root@qe-blade-08 ~]# nisdomainname nis.test

[root@qe-blade-08 ~]# systemctl restart rpcbind

[root@qe-blade-08 ~]# systemctl start ypbind

[root@qe-blade-08 ~]# ps -ef|grep nscd
root     29031 28839  0 21:07 pts/0    00:00:00 grep --color=auto nscd

[root@qe-blade-08 ~]# ps -ef|grep sssd
root     29033 28839  0 21:07 pts/0    00:00:00 grep --color=auto sssd

[root@qe-blade-08 ~]# id ypuser1
uid=964200001(ypuser1) gid=964200001(ypuser1) groups=964200001(ypuser1)

[root@qe-blade-08 ~]# ypcat group|grep ypuser1
ypgroup1:*:964200003:ypuser1
ypuser1:*:964200001:

[root@qe-blade-08 ~]# getent group ypgroup1
ypgroup1:*:964200003:ypuser1

[root@qe-blade-08 ~]# id ypuser1
uid=964200001(ypuser1) gid=964200001(ypuser1) groups=964200001(ypuser1)

[root@qe-blade-08 ~]# ypcat group|grep ypuser1
ypgroup1:*:964200003:ypuser1
ypuser1:*:964200001:

[root@qe-blade-08 ~]# getent group ypgroup1
ypgroup1:*:964200003:ypuser1

Kamil,

I can provide you with hosts where the problem occurs if that helps.  Let me know.

Thanks,
Scott

(In reply to Scott Poore from comment #18)
> I'm not quite sure what is different about your setup but, I'm able to
> reproduce this in a different environment.

I tried exactly the steps from comment #18.  The output was equivalent up to the first invocation of 'id ypuser1' where it already provided the expected result:

# id ypuser1
uid=1331200001(ypuser1) gid=1331200001(ypuser1) groups=1331200001(ypuser1),1331200003(ypgroup1)

(In reply to Scott Poore from comment #19)
> I can provide you with hosts where the problem occurs if that helps.

That would be really appreciated!

I have verified that id(1) correctly prints what getgrouplist(3) returns.  It returns nothing but the group argument it got on its output.  This seems to be inconsistent with the output of getent(1), which says that 'ypuser1' is member of 'ypgroup1':

# getent group ypgroup1
ypgroup1:*:745800003:ypuser1

strace shows that /lib64/libnss_nis.so.2 is not opened upon invocation of getgrouplist(3) despite it 'nis' is enabled in /etc/nsswitch.conf for group lookup:

# grep ^group /etc/nsswitch.conf
group:  files nis

# strace -e trace=open ./a.out 
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 3
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libnss_files.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/etc/group", O_RDONLY|O_CLOEXEC)  = 3
666
+++ exited with 0 +++

On the other hand, getent opens /lib64/libnss_nis.so.2:
# strace -e trace=open getent group ypgroup1
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 3
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libnss_files.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/etc/group", O_RDONLY|O_CLOEXEC)  = 3
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libnss_nis.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libnsl.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/var/yp/binding/nis.test.2", O_RDONLY) = 3
ypgroup1:*:745800003:ypuser1
+++ exited with 0 +++

I am switching the component to glibc...

Created attachment 1190824 [details]
minimal example (lookup only)

# curl -JO 'https://bugzilla.redhat.com/attachment.cgi?id=1190824'

# sh list-groups.c
+ getent group ypgroup1
ypgroup1:*:745800003:ypuser1
+ gcc -Wall -Wextra list-groups.c
+ ./a.out
666
+ exit 0

# service nscd start
Redirecting to /bin/systemctl start  nscd.service

# sh list-groups.c
+ getent group ypgroup1
ypgroup1:*:745800003:ypuser1
+ gcc -Wall -Wextra list-groups.c
+ ./a.out
745800003
666
+ exit 0

# rpm -q glibc
glibc-2.17-156.el7.x86_64

Note that the issue recurs if nscd is stopped again:

# service nscd stop 
Redirecting to /bin/systemctl stop  nscd.service

# sh list-groups.c
+ getent group ypgroup1
ypgroup1:*:745800003:ypuser1
+ gcc -Wall -Wextra list-groups.c
+ ./a.out
666
+ exit 0

From the /etc/nsswitch.conf file of the test client system:

initgroups: files

You need add “nis” there as well, not just on the “group” line.

Arguably, it is an nscd bug that you see NIS data with getgrouplist under such circumstances.

(In reply to Florian Weimer from comment #27)
> From the /etc/nsswitch.conf file of the test client system:
> 
> initgroups: files
> 
> You need add “nis” there as well, not just on the “group” line.

Indeed.  It is documented in the nsswitch.conf(5) man page.  My mistake.

> Arguably, it is an nscd bug that you see NIS data with getgrouplist under
> such circumstances.

That is exactly what made it less obvious I guess...

Yes, I do see that adding nis to initgroups fixes the issue.  Also, commenting it out altogether resolves the issue as well:

# grep initgroups /etc/nsswitch.conf
initgroups: files

# id ypuser1
uid=745800001(ypuser1) gid=745800001(ypuser1) groups=745800001(ypuser1)

# sed -i 's/^.*\(initgroups\)/#\1/' /etc/nsswitch.conf ; grep initgroups /etc/nsswitch.conf
#initgroups: files

# id ypuser1
uid=745800001(ypuser1) gid=745800001(ypuser1) groups=745800001(ypuser1),745800003(ypgroup1)

I went back and checked again on a fresh, minimal RHEL 7.2 install and that had initgroups commented out:

# rpm -q glibc
glibc-2.17-105.el7.x86_64

# grep initgroups /etc/nsswitch.conf
#initgroups: files

So, was there a change in glibc to uncomment that in the latest release?

Thanks,
Scott

(In reply to Scott Poore from comment #31)
> So, was there a change in glibc to uncomment that in the latest release?

The file has not been changed within Red Hat Enterprise Linux 7 on the glibc side.  I have older installations which contain the line “initgroups: files”, too.

Please note that bug 1294574 is quite different from this bug; it is specific to Fedora.  It was not present in glibc 2.17.

I think I may have more information about my issue.  My test was running the following:

authconfig --enablekrb5 --update

That is before the id command did not return all expected groups.

On a RHEL 7.2 server I did see that initgroups was not commented on clean build.  But, I installed IPA and then it was commented (by ipa-server-install).  I uncommented it and then just ran that authconfig command and it was commented again:

[root@master ~]# grep initgroups /etc/nsswitch.conf
#initgroups: files

[root@master ~]# vi /etc/nsswitch.conf

[root@master ~]# grep initgroups /etc/nsswitch.conf
initgroups: files

[root@master ~]# authconfig --enablekrb5 --update

[root@master ~]# grep initgroups /etc/nsswitch.conf
#initgroups: files

In at least some of my tests I noticed later that authconfig was missing.  I modified my tests to specifically install it in case a dependency changed.  I've removed another workaround that runs nscd and I'm running a test to see if everything is clear now from my tests.

This turns out to be a bug in the nscd configuration, we need to employ the same fallback logic as is used in the non-nscd code.

Currently nscd just uses groups data with no initgroups.

We are tracking this bug upstream in this issue:
https://sourceware.org/bugzilla/show_bug.cgi?id=25835

We will be working on the bug upstream, fixing it there, and once the fix is available we can backport to RHEL7 or RHEL8 depending on requirements.

I'm marking this CLOSED/UPSTREAM.