Huh, I wonder what broke between F33 and F34. Maybe https://src.fedoraproject.org/rpms/systemd/c/0e4b90f1130a3da0989e9251b96090f21ddde878?branch=f33 is a little suspicious? That was backported to F33, but it's a post-release update, so it doesn't affect the live image.

It might be caused by 0e4b90f1130a3da0989e9251b96090f21ddde878… This sounds plausible: we changed
systmed to not create the symlink, so NM gets a chance to create its own symlink.

One fix might be to configure NM to not create the symlink. But I think it might be more
reliable to remove the symlink from the image in some final phase. Either way, I don't think
we should force creation of the symlink in systemd.

This is an older post-install Fedora 34 system but has all the latest updates:


[chris@flap ~]$ ls -l /etc/resolv.conf 
-rw-r--r--. 1 root root 291 Feb 28 13:07 /etc/resolv.conf
[chris@flap ~]$ cat /etc/resolv.conf
# Generated by NetworkManager
search hsd1.co.comcast.net flap.local
nameserver 75.75.75.75
nameserver 75.75.76.76
nameserver 2001:558:feed::1
# NOTE: the libc resolver may not support more than 3 nameservers.
# The nameservers listed below may not be recognized.
nameserver 2001:558:feed::2
[chris@flap ~]$ resolvectl 
Global
         Protocols: LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: foreign
Current DNS Server: 75.75.75.75
       DNS Servers: 75.75.75.75 75.75.76.76 2001:558:feed::1 2001:558:feed::2
        DNS Domain: flap.local hsd1.co.comcast.net

Link 2 (wlp108s0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 75.75.75.75
       DNS Servers: 75.75.75.75 75.75.76.76 2001:558:feed::1 2001:558:feed::2
        DNS Domain: hsd1.co.comcast.net


Now do:  rm /etc/resolv.conf; sync; reboot -f  (If I don't reboot fast, or delete it in the initramfs, NM very quickly replaces it before the reboot). And upons reboot things look like this:

[chris@flap ~]$ ls -l /etc/resolv.conf 
lrwxrwxrwx. 1 root root 39 Feb 28 13:21 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
[chris@flap ~]$ resolvectl
Global
       Protocols: LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub

Link 2 (wlp108s0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 75.75.75.75
       DNS Servers: 75.75.75.75 75.75.76.76 2001:558:feed::1 2001:558:feed::2
        DNS Domain: hsd1.co.comcast.net

From the above we can conclude that resolved will win at boot time if there is no resolv.conf file, and it will create a symlink. If a file exists, it won't touch it. But what's the correct behavior for a clean install? Should resolv.conf mode be stub, or foreign?

The /etc/resolv.conf file appears late in the installation, possibly when the chroot is setup to do the bootloader and initramfs stuff. The installer doesn't create resolv.conf, and the base image being copied doesn't contain either resolv.conf file or symlink. Given how aggressive NetworkManager wants to create this file, it makes sense that in the chroot it creates the file.

We could avoid this by installing a NetworkManager configuration file with dns=systemd-resolved. I had been thinking it shouldn't be necessary, but it would be more robust and should probably avoid this issue.

> But what's the correct behavior for a clean install? Should resolv.conf mode be stub, or foreign?

We want the stub mode, of course.

Proposed as a Freeze Exception for 34-beta by Fedora user chrismurphy using the blocker tracking app because:

 This is an unintended regression from Fedora 33, so we'd like to get a fix into beta 34. Again, unclear to me what criterion this could fall under to be a blocker for either beta or final.

> From the above we can conclude that resolved will win at boot time if there is no resolv.conf file, and it will create a symlink.

Yep, that is expected. There's a tmpfiles config and systemd-tmpfiles-create.service runs very early.

> We could avoid this by installing a NetworkManager configuration file with dns=systemd-resolved.

A question though: is network access necessary in the live image and are there any programs which would use the symlink?
(nss-resolve does not need the symlink, and the default config has nss-resolve, but if arbitrary programs
are running, we'd want it.).

Because if we *need* the resolv.conf symlink in the live image, then I think we'd update the systemd
scriptlet to create it, rather then let NM create one.

(In reply to Zbigniew Jędrzejewski-Szmek from comment #6)
> A question though: is network access necessary in the live image

Yes.

> and are
> there any programs which would use the symlink?
> (nss-resolve does not need the symlink, and the default config has
> nss-resolve, but if arbitrary programs
> are running, we'd want it.).

No, I do not care if the symlink is there in the live image. Nothing in the live image should depend on /etc/resolv.conf. If you want to use the live image to run something that does, I think it's OK for it to be broken.

However, that symlink is how NetworkManager knows to switch to mode dns=systemd-resolved. So without the symlink, we need to add a NetworkManager configuration drop-in to force dns=systemd-resolved.
 
> Because if we *need* the resolv.conf symlink in the live image, then I think
> we'd update the systemd
> scriptlet to create it, rather then let NM create one.

I don't think NetworkManager ever creates a symlink? When the symlink does not already exist, it will use mode dns=default and will create its own plain file /etc/resolv.conf. That's the bug here, right? That shouldn't be happening.

Discussed during the 2021-03-01 blocker review meeting: [0]

The decision to classify this bug as an "AcceptedFreezeException (Beta)" was made as it is a noticeable issue that cannot be fixed with an update.

[0] https://meetbot.fedoraproject.org/fedora-blocker-review/2021-03-01/f34-blocker-review.2021-03-01-17.01.txt

(In reply to Zbigniew Jędrzejewski-Szmek from comment #2)
> One fix might be to configure NM to not create the symlink. But I think it
> might be more
> reliable to remove the symlink from the image in some final phase. Either
> way, I don't think
> we should force creation of the symlink in systemd.

NetworkManager *never* creates /etc/resolv.conf as a symlink.

Having the file as a symlink is accepted by NetworkManager an "act of configuration" for who is responsible for that file. That is not NetworkManager's call to make.

Especially with the setting rc-manager=syslink (the effective default), NetworkManager does not touch /etc/resolv.conf if it is a symlink.

So by default, just create /etc/resolv.conf as a symlink, and NetworkManager will stay away.

> I don't think NetworkManager ever creates a symlink? When the symlink does not already exist, it will use mode dns=default and will create its own plain file /etc/resolv.conf. 

> NetworkManager *never* creates /etc/resolv.conf as a symlink.

Yes. Apologies, I had something like "NM creates its own file" in mind, but wrote "symlink". You are both
of course correct that it's not a symlink.

--

But…, how do we solve it. systemd-resolved is running without privileges, so it cannot create the symlink.
I see the following options:
1. add a helper script which creates the symlink before (or after) systemd-resolved is launched
2. distribute a drop-in with NM config with systemd-resolved (if NM config syntax allows that)
3. accept the fact that systemd-resolved is the default and switch the default in NM

I like option 3 the most: it removes a inconsistency in our setup that is clearly causing problems.
The other two solutions are overly complicated for real benefit.

$SOMEBODY makes the choice that systemd-resolved should be used (as default).

That goes beyond `systemctl enable systemd-resolved`, because when configuring the resolver, you may needs `systemctl enable/disable systemd-resolved`, adjust nsswitch.conf, create /etc/resolv.conf as symlink (or plain fail), configure NetworkManager.conf accordingly (which usually means to configure nothing, if you have a standard setup with /etc/resolv.conf symlinking to systemd-resolved).

When installing a Fedora system (or upgrade from F32), we have tools that setup the system. In that case, those installation tools act on behalf of the administrator and are responsible to get this right.

Isn't this something that anaconda should do during the installation from live image? IIUC, anaconda just copies the files from the installation image over, but then afterwards it needs to adjust the bits where the installed system differs from the live system.

(In reply to Zbigniew Jędrzejewski-Szmek from comment #10)
> 2. distribute a drop-in with NM config with systemd-resolved (if NM config
> syntax allows that)

Not only is it allowed, it is easy. As a practical matter, this would be easier than modifying anaconda. But I guess it won't fully solve the problem? Something still has to create the symlink! (Will systemd create the symlink itself if /etc/resolv.conf does not exist?)

Maybe we should start by reverting 0e4b90f1130a3da0989e9251b96090f21ddde878 to see if that gets us back to a non-broken default configuration, since that appears to be what introduced the regression?

>(Will systemd create the symlink itself if /etc/resolv.conf does not exist?)

On normal boot, yes. But when anaconda sets up a chroot to do bootloader+dracut stuff, it seems NetworkManager creates a file because it is present before reboot following an installation; but it is not present prior to the installer's bootloader configuration steps.

Proposed as a Blocker for 34-beta by Fedora user catanzaro using the blocker tracking app because:

 This is a bug in a Critical Path package that:
 * "Cannot be fixed with a future stable update": this is a bug during the install process so it cannot be fixed by an update. The system is installed with an undesired configuration, and we cannot automatically fix the undesired configuration in a future update without hurting users who have intentionally opted out of using systemd-resolved.
 * "Has a severity rating of high or greater and no reasonable workaround." I'm going to be a little loose with my interpretation of "no reasonable workaround." The workaround is actually pretty easy -- delete /etc/resolv.conf and recreate it as a symlink to /run/systemd/resolve/stub-resolve.conf -- but I propose that it's not reasonable to expect everyone installing Fedora 34 to do that manually.

Fedora-Server-dvd-x86_64-34-20210304.n.0.iso
    * Booted ISO's /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
    * Booted ISO's /mnt/sysroot/etc/resolv.conf says it's managed by systemd-resolved
    * Reboot to the installed system /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

Fedora-Everything-netinst-x86_64-34-20210304.n.0.iso
    * Booted ISO's /etc/resolv.conf says it's managed by NetworkManager
    * Booted ISO's /mnt/sysroot/etc/resolv.conf says it's managed by NetworkManager
    * Reboot to the installed system /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

It's weird these differ in the booted ISO environment. How does the end result happen? And why can't that same mechanism work for Live?

Fedora-Workstation-Live-x86_64-34-20210304.n.0.iso (no change from earlier reporting in this bug)
    * Booted ISO's /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
    * Booted ISO's /mnt/sysroot/etc/ has no resolv.conf until install is nearly finished, during "anaconda: progress: Network configuration"
    * Reboot to the installed system /etc/resolv.conf says it's managed by NetworkManager

+4 blocker votes in https://pagure.io/fedora-qa/blocker-review/issue/265 , marking accepted.

So, we need to decide what to actually *do* to address this. Zbigniew, if you don't agree with reverting 0e4b90f1130a3da0989e9251b96090f21ddde878 because of https://bugzilla.redhat.com/show_bug.cgi?id=1891847 , can you please propose / submit an alternative?

Note that Beta Go/No-Go is Thursday. I really need to be able to request a candidate compose today or tomorrow to meet that date, and we can't do a compose without fixing this.

https://github.com/rhinstaller/anaconda/pull/3228 ?

A short summary of findings:
the file is created by Anaconda itself.

The comment in the code says:
> make name resolution work for rpm scripts in chroot
That doesn't seem like a terribly good reason, because scriptlets in general shouldn't touch the network.
But it is possible that there are some other things called from the kickstart that would access the network.

The minimal fix is change the copy function to preserve the symlink. The symlink  is relative, so it looks
broken when resolved from the live image root, but it resolves fine in the chroot under /mnt/sysroot which
has /run mounted. It'll also resolve fine after rebooting into the installed image.

I was wondering whether to completely rip out this code and not create the symlink at all, but that seems
like a much bigger change. If anaconda is invoking things in the chroot, without any resolv.conf name resolution
will not work. (nss-resolve doesn't work because systemd-resolved is not running, and nss-dns needs
/etc/resolv.conf to get a list of servers.)

(In reply to Zbigniew Jędrzejewski-Szmek from comment #19)
> A short summary of findings:
> the file is created by Anaconda itself.
> 
> The comment in the code says:
> > make name resolution work for rpm scripts in chroot
> That doesn't seem like a terribly good reason, because scriptlets in general
> shouldn't touch the network.
> But it is possible that there are some other things called from the
> kickstart that would access the network.

It is hard to say / audit if there is something that requires the name resolution. The original request was in bug 1048520. I wouldn't think anaconda itself requires it but there may be user %post scripts relying on it.
So I'd agree with minimal change here.

(In reply to Chris Murphy from comment #15)

> 
> It's weird these differ in the booted ISO environment. How does the end
> result happen? And why can't that same mechanism work for Live?

Anaconda behaves differently on Live.

> 
> Fedora-Workstation-Live-x86_64-34-20210304.n.0.iso (no change from earlier
> reporting in this bug)
>     * Booted ISO's /etc/resolv.conf ->
> ../run/systemd/resolve/stub-resolv.conf
>     * Booted ISO's /mnt/sysroot/etc/ has no resolv.conf until install is
> nearly finished, during "anaconda: progress: Network configuration"
>     * Reboot to the installed system /etc/resolv.conf says it's managed by
> NetworkManager

The difference compared to Fedora 33 is that the F33 image (rsync-ed to the target system) contains /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf. So on Fedora 33 Anaconda does not overwrite it with /etc/resolv.conf (by copy of target) at the end of installation. On F34 there is no /etc/resolv.conf rsynced to /mnt/sysroot/etc/resolv.conf, so - as Chris described above - Anaconda copies the /etc/resolv.conf target.

So is the link missing on F34 image expected ?

If so, as a fix in anaconda we could use a patch similar to the https://github.com/rhinstaller/anaconda/pull/3228 (if there is a symlink, copy the symlink, not the target) but applied at this place (the end of installation): https://github.com/rhinstaller/anaconda/blob/9a2358fdd3d126ab9925155285d2db4caa6b8b0e/pyanaconda/modules/network/installation.py#L220

> So is the link missing on F34 image expected ?

Yeah. The rpm doesn't have enough information to create the symlink. And in many scenarios,
the symlink simply shouldn't be there. So the symlink is not created by simply installing
the rpms into the sysroot.

Ah yeah, on Lives there's sort of two roots. The "Booted ISO's /etc" is device-mapper live-rw and the resolv.conf symlink is created during boot on the rw overlay by resolved. Whereas the ro live-base is what's used by anaconda when rsync'ing during the installation. And that live-base does not have a symlink, i.e. the image itself does not have a symlink.

(In reply to Radek Vykydal from comment #21)

> If so, as a fix in anaconda we could use a patch similar to the
> https://github.com/rhinstaller/anaconda/pull/3228 (if there is a symlink,
> copy the symlink, not the target) but applied at this place (the end of
> installation):
> https://github.com/rhinstaller/anaconda/blob/
> 9a2358fdd3d126ab9925155285d2db4caa6b8b0e/pyanaconda/modules/network/
> installation.py#L220

https://github.com/rhinstaller/anaconda/pull/3230

On the topic of #c19 and #c20 - yeah, while it's not correct for Fedora package scriptlets to require network access, that's not a requirement on third party packages that someone may be including in an install. And aside from RPM scriptlets, *user-defined kickstart actions* may run in the installed system chroot during install and require network access. Heck, we have a feature for enrolling in a FreeIPA or AD domain via kickstart which may require name resolution to work from the installed system chroot, I think. I don't think we can expect to break name resolution in the installed system chroot in the installer env and not have any issues.

This seems like a complex issue and I'm worried we'll inadvertently break something else, but I'll at least try and run the openQA tests on a scratch build with the proposed fix. That will also give us live and netinst images I can post here for testing.

Hmm. So I'm running the openQA tests on a scratch build at the moment:
https://openqa.stg.fedoraproject.org/tests/overview?build=Kojitask-63440877-NOREPORT&version=34&distri=fedora&groupid=2

the netinst creation test worked fine and I have the ISO it generated, but the live image creation test failed once and looks like it may shortly fail again. That suggests that this change might actually prevent live image creation working, for some reason. I'll try and get more details soon if I can.

hmm, actually now that I look, I'm seeing the same kinda failure for the same test on several other updates too. So probably not caused by this change, at least. I'll just keep re-running the test till it works.

https://www.happyassassin.net/temp/01072636-Fedora-Workstation-Live-x86_64-63440877.iso is the live image openQA built with a scratch build with this anaconda patch included. Please test it. I'll also upload a netinst image shortly.

https://www.happyassassin.net/temp/01072285-63440877-netinst-x86_64.iso is the netinst ISO openQA built, also with the change.

I did a test install of the live image, it succeeded, and on the installed system, /etc/resolv.conf is a symlink to ../run/systemd/resolve/stub-resolv.conf , which cmurf says is what we wanted to happen. So that seems good.

Live image has 
/etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf in the ro live-base, and once installed.

Netinstall image
/etc/resolv.conf is a file 
/mnt/sysroot/etc/resolv.conf  -> ../run/systemd/resolve/stub-resolv.conf   ## this is in the chroot
/etc/resolv.conf  -> ../run/systemd/resolve/stub-resolv.conf    ## reboot to installed


Looks correct unless netinstall image should have symlink for resolvd stub mode in the installation environment.

OK I got Live image reporting wrong in c30. There is no file or symlink on the ro live-base, but it's there once installed. It appears near the end of the installation in /mnt/sysimage/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf and remains following reboot. So I think it's all OK.

Reassigning to anaconda since that's where we're trying to fix this.

FEDORA-2021-4dcfcee13f has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-4dcfcee13f

(In reply to Chris Murphy from comment #30)
 
> Looks correct unless netinstall image should have symlink for resolvd stub
> mode in the installation environment.

That is expected (out of scope of this BZ I'd say), although something we may need to look at.

Regarding the risks of the proposed Anaconda fix (comment #24):

I think it could affect the use cases mentioned in comment #25 only in a very limited way.

The change touches only creating of /mnt/sysimage/etc/resolv.conf at the end of installation - "Network configuration" task which is executed after package installation, but before kickstart %post scripts are run.

But even for the %post scripts we should be safe because the change will effectively change behaviour only in LiveOS installations as in the ISO/rpm installations the /mnt/sysimage/etc/resolv.conf is already replaced during systemd package installation IIUIC [1].
And in LiveOS installations the only case I can think of where the change that could break something would be: /etc/resolv.conf is a symlink for which its target is expected to be copied to installed system which doesn't seem as a real use case to me.

[1] There might be theoretically some cases where the /mnt/sysyimage/etc/resolv.conf copied by Anaconda is not replaced but I have troubles figuring out some - it would probably need to be a symlink created by some kickstart script?

FEDORA-2021-4dcfcee13f has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-4dcfcee13f`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-4dcfcee13f

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Fix is verified in RC1

FEDORA-2021-0be5f28658 has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2021-a12fa51168 has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-a12fa51168

FEDORA-2021-a12fa51168 has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-a12fa51168`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-a12fa51168

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2021-a12fa51168 has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.