Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1811935

Summary: mdns breaks on F31->F32 upgrade
Product: [Fedora] Fedora Reporter: Kamil Páral <kparal>
Component: nss-mdnsAssignee: Adam Goode <adam>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 32CC: adam, awilliam, bugzilla, jhrozek, lpoetter, pbrezina, samuel-rhbugs, Simon.Gerhards
Target Milestone: ---Keywords: CommonBugs
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: https://fedoraproject.org/wiki/Common_F32_bugs#mdns-upgrade-broken
Fixed In Version: nss-mdns-0.14.1-7.fc32 nss-mdns-0.14.1-7.fc30 nss-mdns-0.14.1-7.fc31 nss-mdns-0.14.1-7.el8 nss-mdns-0.14.1-7.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-03-27 08:00:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
F31 nsswitch.conf
none
F31F32 nsswitch.conf.rpmnew
none
post-upgrade logs none

Description Kamil Páral 2020-03-10 09:27:09 UTC
Description of problem:

See the problem reported and discussed in detail here:
https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org/thread/Z3OOSOXFBNM7AXQCHM4HWG2AFAO3OTDB/


$ ssh machine.local

Works on F31. Fails on F32 *upgrade*. Works on F32 clean. 

Best I can tell is something is modifying /etc/nsswitch.conf during the upgrade.

Before dnf system-upgrade (and clean)
hosts:      files mdns4_minimal [NOTFOUND=return] dns myhostname

After dnf system-upgrade (only):
hosts:      files dns myhostname

If I restore this, and restart avahi-daemon.service, the ssh command
works again.

It's not reproducible from a new clean install of F31 without updates.
So there's some update over the past six months that causes a state
change in nsswitch.conf that then results in it being borked upon
system-upgrade.


Version-Release number of selected component (if applicable):
F31:
authselect-1.1-4.fc31.x86_64
F32:
authselect-1.2-1.fc32.x86_64

How reproducible:
happened to multiple people including me

Steps to Reproduce:
to be clarified

Actual results:
mdns configuration is overwritten on upgrade

Expected results:
mdns configuration is kept on upgrade

Comment 1 Pavel Březina 2020-03-10 11:23:29 UTC
Can you please attach /etc/nsswitch.conf, /etc/nsswitch.conf.bak and /etc/nsswitch.conf.rpmnew and /etc/authselect/user-nsswitch.conf?

Also what is the output of 'authselect check' and 'authselect current'?

Thank you.

Comment 2 Kamil Páral 2020-03-10 11:27:59 UTC
Chris, do you still have those files? Because I got rid of them. If you don't, I'll try to reproduce the issue in a VM.

Comment 3 Pavel Březina 2020-03-10 12:15:08 UTC
If you don't have them, I can try to reproduce it first. If I understand it correctly it should be just about installing nss-mdns on F31 and they upgrade, right?

Comment 4 Kamil Páral 2020-03-10 12:42:52 UTC
My best guess at reproducing this is to install F31 from Live, make sure machine.local resolving works, fully update, make sure machine.local resolving works, upgrade to F32, see that machine.local resolving is broken. If you can't test it today or Chris can't provide logs, I'll work on reproducing this and providing logs tomorrow.

Comment 5 Chris Murphy 2020-03-10 14:51:53 UTC
Created attachment 1668971 [details]
F31 nsswitch.conf

This is the nsswitch.conf file immediately prior to F31->F32 upgrade.

After the upgrade to F32:
- hosts:      files mdns4_minimal [NOTFOUND=return] dns myhostname
+ hosts:      files dns myhostname

Reverting that difference manually fixed the problem.

Comment 6 Chris Murphy 2020-03-10 14:59:24 UTC
I cannot reproduce the problem in a VM doing this:

1. Clean install Fedora-Workstation-Live-x86_64-31-1.9.iso (official GA release)
2. Reboot, update only dnf, et al and install system-upgrade plugin
3. dnf system-upgrade download and reboot

Comment 7 Chris Murphy 2020-03-10 15:11:46 UTC
Regarding the original report, it's a baremetal system, clean install (not an upgrade from Fedora 39 or 30); but must have been prerelease, which explains why the nsswitch.conf line 1 time stamp does not match between baremetal and VM.

Comment 8 Chris Murphy 2020-03-10 15:15:42 UTC
Created attachment 1668974 [details]
F31F32 nsswitch.conf.rpmnew

Following successful F31->F32 upgrade (baremetal, original problem report computer), there is '/etc/nsswitch.conf.rpmnew

Attaching it.

Comment 9 Chris Murphy 2020-03-10 17:48:47 UTC
The first file is post-upgrade F31->F32; the second is from a snapshot of this system prior to the upgrade (while it was still F31). This is very suspicious as the source of the problem I experienced.

$ diff /etc/authselect/user-nsswitch.conf root.fc31.20200302/etc/authselect/user-nsswitch.conf
40c40
< hosts:      files dns myhostname
---
> hosts:      files mdns4_minimal [NOTFOUND=return] dns myhostname
$

Comment 10 Kamil Páral 2020-03-10 18:11:16 UTC
(In reply to Kamil Páral from comment #4)
> My best guess at reproducing this is to install F31 from Live, make sure
> machine.local resolving works, fully update, make sure machine.local
> resolving works, upgrade to F32, see that machine.local resolving is broken.

I've just reproduced the problem with exactly these steps (in a VM). I'll attach the logs.

In F32:

$ sudo authselect check
Current configuration is valid.

$ sudo authselect current
Profile ID: sssd
Enabled features:
- with-fingerprint
- with-silent-lastlog

Comment 11 Kamil Páral 2020-03-10 18:15:15 UTC
Created attachment 1669025 [details]
post-upgrade logs

Comment 12 Chris Murphy 2020-03-10 19:16:11 UTC
These are from clean installs from the following media.
media/1 = Fedora-Workstation-Live-x86_64-31-1.9.iso
media/2 = Fedora-Workstation-Live-x86_64-32-20200308.n.0.iso



# diff /media/1/root.clean/etc/authselect/user-nsswitch.conf /media/2/root.clean/etc/authselect/user-nsswitch.conf 
4c4
< # An example Name Service Switch config file. This file should be
---
> # Name Service Switch config file. This file should be
7,11c7,9
< # The entry '[NOTFOUND=return]' means that the search for an
< # entry should stop if the search in the previous entry turned
< # up nothing. Note that if the search failed due to some other reason
< # (like no NIS server responding) then the search continues with the
< # next entry.
---
> # Valid databases are: aliases, ethers, group, gshadow, hosts,
> # initgroups, netgroup, networks, passwd, protocols, publickey,
> # rpc, services, and shadow.
13c11
< # Valid entries include:
---
> # Valid service provider entries include (in alphabetical order):
15,16c13,14
< #	nisplus			Use NIS+ (NIS version 3)
< #	nis			Use NIS (NIS version 2), also called YP
---
> #	compat			Use /etc files plus *_compat pseudo-db
> #	db			Use the pre-processed /var/db files
19,20d16
< #	db			Use the pre-processed /var/db files
< #	compat			Use /etc files plus *_compat pseudo-databases
22,23c18,19
< #	sss			Use sssd (System Security Services Daemon)
< #	[NOTFOUND=return]	Stop searching if not found so far
---
> #	nis			Use NIS (NIS version 2), also called YP
> #	nisplus			Use NIS+ (NIS version 3)
25,26c21
< # 'sssd' performs its own 'files'-based caching, so it should
< # generally come before 'files'.
---
> # See `info libc 'NSS Basics'` for more information.
28,33c23,49
< # WARNING: Running nscd with a secondary caching service like sssd may lead to
< # 	   unexpected behaviour, especially with how long entries are cached.
< 
< # To use 'db', install the nss_db package, and put the 'db' in front
< # of 'files' for entries you want to be looked up first in the
< # databases, like this:
---
> # Commonly used alternative service providers (may need installation):
> #
> #	ldap			Use LDAP directory server
> #	myhostname		Use systemd host names
> #	mymachines		Use systemd machine names
> #	mdns*, mdns*_minimal	Use Avahi mDNS/DNS-SD
> #	resolve			Use systemd resolved resolver
> #	sss			Use System Security Services Daemon (sssd)
> #	systemd			Use systemd for dynamic user option
> #	winbind			Use Samba winbind support
> #	wins			Use Samba wins support
> #	wrapper			Use wrapper module for testing
> #
> # Notes:
> #
> # 'sssd' performs its own 'files'-based caching, so it should generally
> # come before 'files'.
> #
> # WARNING: Running nscd with a secondary caching service like sssd may
> # 	   lead to unexpected behaviour, especially with how long
> # 	   entries are cached.
> #
> # Installation instructions:
> #
> # To use 'db', install the appropriate package(s) (provide 'makedb' and
> # libnss_db.so.*), and place the 'db' in front of 'files' for entries
> # you want to be looked up first in the databases, like this:
38a55
> # In order of likelihood of use to accelerate lookup.
42d58
< 
44,51d59
< 
< bootparams: files
< 
< ethers:     files
< netmasks:   files
< networks:   files
< protocols:  files
< rpc:        files
53d60
< 
55,57d61
< 
< publickey:  files
< 
58a63
> 
59a65,72
> ethers:     files
> gshadow:    files
> # Allow initgroups to default to the setting for group.
> # initgroups: files
> networks:   files dns
> protocols:  files
> publickey:  files
> rpc:        files
# 


A simpler summary:

Both files contain these:
passwd:      sss files systemd
shadow:     files sss
group:       sss files systemd
hosts:      files mdns4_minimal [NOTFOUND=return] dns myhostname
services:   files sss
netgroup:   sss
automount:  files sss
aliases:    files
ethers:     files
protocols:  files
publickey:  files
rpc:        files

F31 contains:

networks:   files
bootparams: files
netmasks:   files

F32 contains:

networks:   files dns
gshadow:    files

Comment 13 Pavel Březina 2020-03-12 10:25:49 UTC
I will spin up a custom vm and try to reproduce. I suspect there is an issue in nss-mdns upgrade path.

Here is a little explanations of the different files you see:
- /etc/nsswitch.conf.rpmnew is totally expected - it is a configuration file provided by glibc, upgrade will not change the file but create .rpmnew
- /etc/authselect/user-nsswitch.conf - this is created as a copy of /etc/nsswitch.conf upon authselect installation so the contents might be different for F31-32 upgrade and fresh F32 install
- nss-mdns is used to touch /etc/nsswitch.conf to enable itself on the system, I worked with the maintainer to make this compatible with both non-authselect and authselect configurations. Unfortunately the changes landed in F31 just two months ago so it is possible that it is causing some troubles if it was originally installed without authselect-aware scriptlet.

I will check the upgrade path for both authselect-aware nss-mdns and the older version to see where the problem is.

To work around this issue, please do these steps:
1) Change /etc/authselect/user-nsswitch.conf to contain:
hosts: files mdns4_minimal [NOTFOUND=return] dns myhostname
2) Apply changes
$ sudo authselect apply-changes

Or if you choose to opt out of authselect, you can just edit /etc/nsswitch.conf directly.

Comment 14 Pavel Březina 2020-03-17 11:54:42 UTC
I can reproduce it even with latest nss-mdns in F31 installed directly.

1. Boot up fresh F31
2. Install nss-mdns-0.14.1-5.fc31.x86_64.rpm (authselect fix included)
   mdns is correctly enabled in /etc/authselect/user-nsswitch.conf and propagated to /etc/nsswitch.conf
   /etc/authselect/user-nsswitch.conf.bak is created
3. Upgrade to F32
4. mdns is not configured

/etc/authselect/user-nsswitch.conf was copied to /etc/authselect/user-nsswitch.conf.bak and mdns was removed from user-nsswitch.conf. I will check nss-mdns scriptlets to see what is causing it.

Comment 15 Pavel Březina 2020-03-17 15:45:11 UTC
The problem is that %preun which removes mdns from nsswitch.conf is run also for upgrade path:
https://src.fedoraproject.org/rpms/nss-mdns/blob/master/f/nss-mdns.spec#_65

Switching to nss-mdns, I will open a pull request against it.

Comment 17 Fedora Update System 2020-03-19 13:59:33 UTC
FEDORA-2020-e8df029f18 has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-e8df029f18

Comment 18 Fedora Update System 2020-03-19 13:59:56 UTC
FEDORA-2020-5d6d7d1815 has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2020-5d6d7d1815

Comment 19 Fedora Update System 2020-03-19 14:00:04 UTC
FEDORA-2020-75a4a1e132 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2020-75a4a1e132

Comment 20 Fedora Update System 2020-03-19 14:00:26 UTC
FEDORA-EPEL-2020-b03097f59d has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-b03097f59d

Comment 21 Fedora Update System 2020-03-19 14:00:54 UTC
FEDORA-EPEL-2020-9b0fe95016 has been submitted as an update to Fedora EPEL 8. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-9b0fe95016

Comment 22 Fedora Update System 2020-03-20 01:59:01 UTC
nss-mdns-0.14.1-7.fc32 has been pushed to the Fedora 32 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-e8df029f18

Comment 23 Fedora Update System 2020-03-20 02:13:32 UTC
nss-mdns-0.14.1-7.el8 has been pushed to the Fedora EPEL 8 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-9b0fe95016

Comment 24 Fedora Update System 2020-03-20 02:14:22 UTC
nss-mdns-0.14.1-7.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-b03097f59d

Comment 25 Fedora Update System 2020-03-20 03:08:27 UTC
nss-mdns-0.14.1-7.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-75a4a1e132

Comment 26 Fedora Update System 2020-03-20 03:12:27 UTC
nss-mdns-0.14.1-7.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-5d6d7d1815

Comment 27 Kamil Páral 2020-03-20 12:04:41 UTC
I tested upgrade F31->F32 according to comment 10 and machine.local queries still work even after upgrade. mdns is included in /etc/nsswitch.conf. Looks fixed, thanks.

Comment 28 Fedora Update System 2020-03-27 08:00:53 UTC
FEDORA-2020-e8df029f18 has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 29 Fedora Update System 2020-03-27 10:42:56 UTC
FEDORA-2020-75a4a1e132 has been pushed to the Fedora 30 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 30 Fedora Update System 2020-03-27 13:09:07 UTC
FEDORA-2020-5d6d7d1815 has been pushed to the Fedora 31 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 31 Fedora Update System 2020-04-04 00:31:31 UTC
FEDORA-EPEL-2020-9b0fe95016 has been pushed to the Fedora EPEL 8 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 32 Fedora Update System 2020-04-04 00:43:55 UTC
FEDORA-EPEL-2020-b03097f59d has been pushed to the Fedora EPEL 7 stable repository.
If problem still persists, please make note of it in this bug report.