Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 2074083 - dasbus.error.DBusError: [Errno 2] No such file or directory: '/mnt/sysroot/etc/resolv.conf'
Summary: dasbus.error.DBusError: [Errno 2] No such file or directory: '/mnt/sysroot/et...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: 35
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: systemd-maint
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 2074907 2075612 2077582 2077890 2078950 2080455 2081020 2083411 2083866 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-04-11 13:46 UTC by Jan Pazdziora
Modified: 2022-06-22 21:40 UTC (History)
46 users (show)

Fixed In Version: systemd-249.11-2.fc35 systemd-249.12-3.fc35 systemd-249.12-5.fc35
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-06-21 01:08:18 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Jan Pazdziora 2022-04-11 13:46:22 UTC
Description of problem:

The systemd-249.11-1.fc35 causes anaconda to traceback when @core group is being installed.

Version-Release number of selected component (if applicable):

systemd-249.11-1.fc35

How reproducible:

Deterministic.

Steps to Reproduce:
1. Try to provision Fedora 35 with the default (just @core) package set.

Actual results:

Console output ends with

An unknown error has occured, look at the /tmp/anaconda-tb* file(s) for more details  

Anaconda log ends with

12:50:47,271 CRT exception: Traceback (most recent call last):

  File "/usr/lib64/python3.10/site-packages/pyanaconda/threading.py", line 275, in run
    threading.Thread.run(self)

  File "/usr/lib64/python3.10/threading.py", line 946, in run
    self._target(*self._args, **self._kwargs)

  File "/usr/lib64/python3.10/site-packages/pyanaconda/installation.py", line 415, in run_installation
    queue.start()

  File "/usr/lib64/python3.10/site-packages/pyanaconda/installation_tasks.py", line 311, in start
    item.start()

  File "/usr/lib64/python3.10/site-packages/pyanaconda/installation_tasks.py", line 311, in start
    item.start()

  File "/usr/lib64/python3.10/site-packages/pyanaconda/installation_tasks.py", line 311, in start
    item.start()

  File "/usr/lib64/python3.10/site-packages/pyanaconda/installation_tasks.py", line 496, in start
    self.run_task()

  File "/usr/lib64/python3.10/site-packages/pyanaconda/installation_tasks.py", line 458, in run_task
    self._task(*self._task_args, **self._task_kwargs)

  File "/usr/lib64/python3.10/site-packages/pyanaconda/network.py", line 293, in write_configuration
    sync_run_task(task_proxy)

  File "/usr/lib64/python3.10/site-packages/pyanaconda/modules/common/task/__init__.py", line 46, in sync_run_task
    task_proxy.Finish()

  File "/usr/lib/python3.10/site-packages/dasbus/client/handler.py", line 444, in _call_method
    return self._get_method_reply(

  File "/usr/lib/python3.10/site-packages/dasbus/client/handler.py", line 477, in _get_method_reply
    return self._handle_method_error(error)

  File "/usr/lib/python3.10/site-packages/dasbus/client/handler.py", line 497, in _handle_method_error
    raise exception from None

dasbus.error.DBusError: [Errno 2] No such file or directory: '/mnt/sysroot/etc/resolv.conf'

12:50:47,273 DBG exception: Gtk cannot be initialized
12:50:47,273 DBG exception: In the main thread, running exception handler
12:51:02,933 INF core.util: Running kickstart %%onerror script(s)
12:51:14,215 INF core.util: All kickstart %%onerror script(s) have been run
12:51:14,215 INF kickstart.script: Running kickstart %%traceback script(s)
12:51:14,215 INF kickstart.script: All kickstart %%traceback script(s) have been run
12:51:14,215 INF core.util: Reporting the IPMI event: 10

Expected results:

No traceback, Fedora 35 installs.

Additional info:

This is a regression against systemd-249.9-1.fc35.

Comment 2 Marcos Mello 2022-04-11 23:52:15 UTC
Please raise the priority, as it affects F35 netinstalls... the current Fedora *stable* release.

Comment 3 Zbigniew Jędrzejewski-Szmek 2022-04-12 07:41:03 UTC
*** Bug 2074122 has been marked as a duplicate of this bug. ***

Comment 4 Zbigniew Jędrzejewski-Szmek 2022-04-12 07:41:52 UTC
From the other bug:

During a routine upgrade, systemd-resolved printed this :

  Running scriptlet: systemd-resolved-249.11-1.fc35.x86_64                69/69 
'/etc/resolv.conf' -> '../run/systemd/resolve/stub-resolv.conf'

but without creating /run/systemd/resolve/stub-resolv.conf -- probably because
the directory /run/systemd/resolve does not exist. Result : loss of file /etc/resolv.conf.
  I don't need systemd-resolved. Perhaps systemd-resolved should interpret the absence of /run/systemd/resolve as a cue to abstain.

Comment 5 Zbigniew Jędrzejewski-Szmek 2022-04-12 08:09:50 UTC
OK, I know what is happening. There were two patches applied to Anaconda (https://github.com/rhinstaller/anaconda/pull/3884),
but apparently they didn't make it into F35, but only into F36.

@rvykydal what should we do here? Also pull in the patches for Anaconda or revert that change
in systemd? I guess the later is the correct action. I'll start a build with that chunk reverted.

Comment 6 Jan Pazdziora 2022-04-12 09:05:44 UTC
I'd agree that systemd needs to change/revert. We cannot rely on any changes to anaconda -- Fedora 35 needs to be able to provision with the GA ISOs (and updates repository enabled).

Comment 7 Fedora Update System 2022-04-12 09:06:01 UTC
FEDORA-2022-1bf06aa328 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-1bf06aa328

Comment 8 Radek Vykydal 2022-04-12 11:35:37 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #5)
> OK, I know what is happening. There were two patches applied to Anaconda
> (https://github.com/rhinstaller/anaconda/pull/3884),
> but apparently they didn't make it into F35, but only into F36.
> 
> @rvykydal what should we do here? Also pull in the patches for
> Anaconda or revert that change
> in systemd? I guess the later is the correct action. I'll start a build with
> that chunk reverted.

Unfortunately, yes we need the revert the change. Jan is right in comment #6.

Comment 9 Radek Vykydal 2022-04-12 11:40:16 UTC
To explain, we added the fix to F35 anaconda for eventual respins but the fix didn't make it to F35 GOLD installer (https://bugzilla.redhat.com/show_bug.cgi?id=2019579#c30)

Comment 10 Brian J. Murrell 2022-04-12 12:46:47 UTC
Where can the progress of the revert be tracked so that we know when to remove work-arounds in kickstart installs?

Comment 11 Jan Pazdziora 2022-04-12 12:56:13 UTC
Since this bugzilla is linked to the errata with the build with the fix (see comment 7), status of this bugzilla will get updated accordingly as the errata becomes available in testing and then in stable repo.

Comment 12 Radek Vykydal 2022-04-13 12:49:46 UTC
*** Bug 2074907 has been marked as a duplicate of this bug. ***

Comment 13 Fedora Update System 2022-04-13 15:18:48 UTC
FEDORA-2022-1bf06aa328 has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-1bf06aa328`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-1bf06aa328

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 14 Fedora Update System 2022-04-14 16:40:23 UTC
FEDORA-2022-1bf06aa328 has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-1bf06aa328`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-1bf06aa328

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 15 Jason Tibbitts 2022-04-18 19:29:37 UTC
So the fixed build never made it to stable, because it seems that it broke something else (also involving the problematic symlink).

In the meantime I'm installing F35 without updates enabled, which is a bit suboptimal.  Is there any better workaround?  I could put the unpushed update in a local repository but it was unpushed for a reason....

Comment 16 Marcos Mello 2022-04-19 18:38:49 UTC
systemd-249.11-2.fc35 new bug:

https://bugzilla.redhat.com/show_bug.cgi?id=2075428

Comment 17 Brian Lane 2022-04-20 21:45:17 UTC
*** Bug 2075612 has been marked as a duplicate of this bug. ***

Comment 18 Brian Lane 2022-04-21 17:23:54 UTC
*** Bug 2077582 has been marked as a duplicate of this bug. ***

Comment 19 Radek Vykydal 2022-04-22 14:13:45 UTC
*** Bug 2077890 has been marked as a duplicate of this bug. ***

Comment 20 Fedora Update System 2022-04-23 19:27:40 UTC
FEDORA-2022-1bf06aa328 has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 21 Jan Pazdziora 2022-04-26 12:17:16 UTC
I don't see this issue fixed with systemd-249.11-2.fc35.x86_64. The installation still fails with

dasbus.error.DBusError: [Errno 2] No such file or directory: '/mnt/sysroot/etc/resolv.conf'

Comment 23 PaulB 2022-04-27 15:01:09 UTC
*** Bug 2078950 has been marked as a duplicate of this bug. ***

Comment 24 John Villalovos 2022-04-27 15:22:18 UTC
Trying to run some tests on Fedora 35 and I am still seeing this issue on Fedora 35 network installs.

Comment 25 Zbigniew Jędrzejewski-Szmek 2022-04-28 15:58:41 UTC
That is very strange. I updated the scriptlet to be a bit more robust. Let's see if this next
version works better.

Comment 26 PaulB 2022-04-28 16:06:41 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #25)
> That is very strange. I updated the scriptlet to be a bit more robust. Let's
> see if this next
> version works better.

Zbigniew,
What version of systemd has the fix?

Comment 27 Zbigniew Jędrzejewski-Szmek 2022-04-28 17:20:01 UTC
That'll most likely be systemd-249.12-1.fc35.

Comment 28 Fedora Update System 2022-04-28 20:41:45 UTC
FEDORA-2022-01079468a3 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-01079468a3

Comment 29 Adam Williamson 2022-04-28 23:02:44 UTC
The systemd update is failing openQA tests. It seems like trying to build a Workstation live image with this systemd included fails, due to scriptlet errors in systemd-resolved:

18:53:08,525 INF dnf.rpm: /var/tmp/rpm-tmp.jzmyWP: line 18: mkdir: command not found
/var/tmp/rpm-tmp.jzmyWP: line 19: /var/lib/rpm-state/systemd/systemd-resolved-initial-installation: No such file or directory
warning: %post(systemd-resolved-249.12-1.fc35.x86_64) scriptlet failed, exit status 1

18:53:08,528 ERR dnf.rpm: Error in POSTIN scriptlet in rpm package systemd-resolved
18:53:08,680 INF dnf.rpm: /var/tmp/rpm-tmp.xOgK8A: line 2: ln: command not found

Comment 30 Adam Williamson 2022-04-28 23:07:38 UTC
I think that last line's a stray, sorry. The problem is that the attempt to fix things here added an `mkdir` call to systemd-resolved %post, but systemd-resolved does not Requires(post): coreutils . It probably needs to. Just hope that doesn't cause any circular dependencies.

Comment 31 Zbigniew Jędrzejewski-Szmek 2022-04-29 06:59:17 UTC
Oh right. I'm doing a build without 'ls' or 'mkdir' now, pure 'sh'. If that doesn't work,
the next step will be to switch to %lua. (I  don't want to add a dependency on coreutils
because even if that works here it might cause dependency loops in other scenarios, and
it's hard to figure out what might break.)

Comment 32 Fedora Update System 2022-04-29 08:09:09 UTC
FEDORA-2022-01079468a3 has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-01079468a3`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-01079468a3

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 33 Edgar Hoch 2022-04-29 15:04:40 UTC
systemd 249.12-2.fc35 still fails during Fedora 35 installation.

I have tried a kickstart installation for Fedora 35 using an additional local repository where I have downloaded the packages systemd 249.12-2.fc35 from koji build 1957690.

After installation of the packages the installation stops. anaconda.log contains the following error messages:


16:00:12,348 INF threading: Thread Failed: AnaInstallThread (140677780940352)
16:00:12,348 DBG exception: running handleException
16:00:12,356 CRT exception: Traceback (most recent call last):

  File "/usr/lib64/python3.10/site-packages/pyanaconda/threading.py", line 275, in run
    threading.Thread.run(self)

  File "/usr/lib64/python3.10/threading.py", line 946, in run
    self._target(*self._args, **self._kwargs)

  File "/usr/lib64/python3.10/site-packages/pyanaconda/installation.py", line 415, in run_installation
    queue.start()

  File "/usr/lib64/python3.10/site-packages/pyanaconda/installation_tasks.py", line 311, in start
    item.start()

  File "/usr/lib64/python3.10/site-packages/pyanaconda/installation_tasks.py", line 311, in start
    item.start()

  File "/usr/lib64/python3.10/site-packages/pyanaconda/installation_tasks.py", line 311, in start
    item.start()

  File "/usr/lib64/python3.10/site-packages/pyanaconda/installation_tasks.py", line 496, in start
    self.run_task()

  File "/usr/lib64/python3.10/site-packages/pyanaconda/installation_tasks.py", line 458, in run_task
    self._task(*self._task_args, **self._task_kwargs)

  File "/usr/lib64/python3.10/site-packages/pyanaconda/network.py", line 293, in write_configuration
    sync_run_task(task_proxy)

  File "/usr/lib64/python3.10/site-packages/pyanaconda/modules/common/task/__init__.py", line 46, in sync_run_task
    task_proxy.Finish()

  File "/usr/lib/python3.10/site-packages/dasbus/client/handler.py", line 444, in _call_method
    return self._get_method_reply(

  File "/usr/lib/python3.10/site-packages/dasbus/client/handler.py", line 477, in _get_method_reply
    return self._handle_method_error(error)

  File "/usr/lib/python3.10/site-packages/dasbus/client/handler.py", line 497, in _handle_method_error
    raise exception from None

dasbus.error.DBusError: [Errno 2] Datei oder Verzeichnis nicht gefunden: '/mnt/sysroot/etc/resolv.conf'

16:00:12,359 DBG exception: Gtk running, queuing exception handler to the main loop
16:00:12,359 INF threading: Thread Done: AnaInstallThread (140677780940352)

Comment 34 Zbigniew Jędrzejewski-Szmek 2022-04-29 15:51:45 UTC
I assume that "No such file or directory" is reported by trying to open a dangling symlink.

Let's summarize:
With systemd-249.9-1.fc35, we had #2032085 — Some variants are missing /etc/resolv.conf symlink,
which actually meant that a static file would be created by NetworkManager and copied by Anaconda into the installed image.
So in systemd-249.11-1.fc35, the symlink started being created by systemd-resolved scriptlets during installation.
But this confused Anaconda, and Anaconda got patches in F36 to not try to handle /etc/resolv.conf itself.
Then there were bugs in the scriptlets, and issues with dependencies being underspecified for scriplets, and dependency loops, and rpm not installing dependencies even when they are specified correctly and could be installed, in short a big mess that took a while to pound into shape.
With systemd-249.12-2.fc35, the symlink is correctly created in all circumstances, but Anaconda gets confused because the symlink in the chroot is dangling because it's missing the patches to not touch it, and we can't update Anaconda.

We could create an *absolute* symlink instead: Anaconda would try to read the symlink and load the file from the "host", i.e. it's livecd environment. But this falls flat because systemd-resolved is not installed in the F35 live image :(

I don't have any more nice ideas. I'll try to implement a brute-force approach:
a one time service to remove static /etc/resolv.conf that is created by NetworkManager after reboot.

If anyone else has better ideas, please speak up.

Comment 35 沙包妖梦 2022-04-29 16:53:41 UTC
*** Bug 2080455 has been marked as a duplicate of this bug. ***

Comment 36 沙包妖梦 2022-04-29 17:03:23 UTC
I will delete NetworkManager after first boot (if it's installed). Then use systemd-networkd and systemd-resolved. So I don't want Anaconda generate any network setting for me. Leave it here (a dangling symlink) is the best result for me.

Comment 37 Zbigniew Jędrzejewski-Szmek 2022-04-29 17:38:05 UTC
I updated the update with another build. This time a link ../usr/lib/systemd/resolv.conf
instead of ../run/systemd/resolve/resolv.conf is created, so Anaconda should be happy.
Most users don't configure own resolvers, so they shouldn't care about the difference.

Comment 38 Zbigniew Jędrzejewski-Szmek 2022-04-29 17:40:08 UTC
If people who tested the previous iterations could also test this, that'd be great.

I'm going away until Tuesday (Labour Day and Constitution Day here in Poland), and I'm not taking my computer with me, so I won't work on this for a few days.

Comment 39 沙包妖梦 2022-04-29 17:52:02 UTC
Just test again in my VM:

You can create empty /mnt/sysroot/run/systemd/resolve/stub-resolv.conf file (need mkdir run/systemd/resolve).
Because systemd-resolved is enabled by default, it will update this file with right content after reboot. NetworkManager is not required here.

Comment 40 Zbigniew Jędrzejewski-Szmek 2022-04-29 18:24:30 UTC
(In reply to 沙包妖梦 from comment #39)
> You can create empty /mnt/sysroot/run/systemd/resolve/stub-resolv.conf file
> (need mkdir run/systemd/resolve).
> Because systemd-resolved is enabled by default, it will update this file
> with right content after reboot. NetworkManager is not required here.

I guess that'd work too. I'm not sure if /run is mounted in the installroot. If it is,
then that'd actually work pretty nicely, because the file would go away automatically.
If it's not, and we created the file, we would get a warning after reboot about /run containing
files. We could mount /run in the installroot and then create the directories and file,
but that becomes quite complex.

So let's see if the current approach works… If it does, maybe we can iterate on it with
your idea.

Comment 41 カラム 2022-04-30 01:05:12 UTC
Hi There,

I am still having this issue as of this morning using both Netinstall and full install.

Are there any workaround for this on a clean install?

Should I just use F36 instead?

What are the recommendations here, I need to install ASAP.

Many thanks in advance

P.S. If this is not the right place for this message please advise, I am new at this. Thank you!

Comment 42 Alex Regan 2022-04-30 13:47:47 UTC
I'm also having this problem when attempting to install in a VM at the OVHCloud provider. They don't have a kickstart install option, only "mount ISO", so how do I do a proper install?

If the new anaconda package was released yesterday, why doesn't "install from local mirror" pick it up?

Is there a way to fix this in an alternate terminal during install? Perhaps even download and install the updated anaconda package after it fails at the end of the install?

Comment 43 Dave 2022-04-30 15:53:03 UTC
I solved this problem by doing the following:

- Set up the install as normal (choose disk layout, software, network setup, etc)
- Immediately after clicking "begin installation", switch over to tty3 (ctrl-alt F3)
- Run "df -h" until you see that /mnt/sysroot is mounted
- Run "cd /mnt/sysroot; mkdir -p run/systemd/resolve; touch run/systemd/resolve/stub-resolv.conf; cd /"

You can then switch back to tty6 to watch the install finish successfully.

Comment 44 Edgar Hoch 2022-04-30 16:01:34 UTC
I have still the same error with systemd-249.12-3.fc35 .

Comment 45 Sarim Khan 2022-04-30 16:11:08 UTC
My workaround was to install using fedora server dvd iso, I unplugged ethernet cable before install. Then installation completed successfully. Afterward connected ethernet, configured internet and applied the updates.
Couldn't net netinst iso to work. Off topic: The patch mentioned in other thread, If I apply using inst.updates= boot flag, installer fails to start in bare metal hardware. Worked in VM though :/

Comment 46 Edgar Hoch 2022-04-30 17:43:21 UTC
(In reply to Sarim Khan from comment #45)
> My workaround was to install using fedora server dvd iso, I unplugged
> ethernet cable before install. Then installation completed successfully.
> Afterward connected ethernet, configured internet and applied the updates.

You don't need to unplug the ethernet cable. Just don't include updates repository in the kickstart configuration. For example, comment it:
#repo --name=updates

Comment 47 Fedora Update System 2022-04-30 18:21:29 UTC
FEDORA-2022-01079468a3 has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-01079468a3`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-01079468a3

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 48 Jan Pazdziora 2022-05-01 07:28:24 UTC
(In reply to Alex Regan from comment #42)
> 
> If the new anaconda package was released yesterday, why doesn't "install
> from local mirror" pick it up?

Because anaconda which executes is anaconda on that ISO, not anaconda on the mirror. On a typical installation, anaconda in the repositories does not play any role.

Comment 49 Radek Vykydal 2022-05-04 07:55:56 UTC
*** Bug 2081020 has been marked as a duplicate of this bug. ***

Comment 50 Jan Pazdziora 2022-05-06 13:23:38 UTC
Provisioning Fedora 35 is still broken with updates-testing repo enabled and systemd-249.12-3.fc35.x86_64 installed from it.

Comment 52 Fedora Update System 2022-05-22 01:23:07 UTC
FEDORA-2022-01079468a3 has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 53 PaulB 2022-05-25 13:41:19 UTC
(In reply to Fedora Update System from comment #52)
> FEDORA-2022-01079468a3 has been pushed to the Fedora 35 stable repository.
> If problem still persists, please make note of it in this bug report.

All,
This issue persists.
tested today with Fedora-35 Everything aarch64

---%<-snip->%---
Verifying systemd-libs.aarch64 (361/379)
Verifying systemd-networkd.aarch64 (362/379)
Verifying systemd-oomd-defaults.noarch (363/379)
Verifying systemd-pam.aarch64 (364/379)
Verifying systemd-resolved.aarch64 (365/379)
Verifying systemd-udev.aarch64 (366/379)
Verifying tzdata.noarch (367/379)
Verifying udisks2.aarch64 (368/379)
Verifying util-linux.aarch64 (369/379)
Verifying util-linux-core.aarch64 (370/379)
Verifying vim-data.noarch (371/379)
Verifying vim-minimal.aarch64 (372/379)
Verifying whois-nls.noarch (373/379)
Verifying xz.aarch64 (374/379)
Verifying xz-libs.aarch64 (375/379)
Verifying yum.noarch (376/379)
Verifying zchunk-libs.aarch64 (377/379)
Verifying zram-generator.aarch64 (378/379)
Verifying zram-generator-defaults.noarch (379/379)
.

Performing post-installation setup tasks
.

Configuring installed system
.
.
.
.
.
.
.
.
.
.
.
.
.

Writing network configuration

An unknown error has occured, look at the /tmp/anaconda-tb* file(s) for more details


ine 496, in start
    self.run_task()
  File "/usr/lib64/python3.10/site-packages/pyanaconda/installationaconda/installages/pyanaconda/installation_tasks.py", line 311, in start
    item.start()
  File "/usr/lib64/python3.10/site-packages/pyanaconda/installation.py", line 415, in run_installation
    queue.start()
  File "/usr/lib64/python3.10/threading.py", line 946, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib64/python3.10/site-packages/pyanaconda/threading.py", line 275, in run
    threading.Thread.run(self)
dasbus.error.DBusError: [Errno 2] No such file or directory: '/mnt/sysroot/etc/resolv.conf'

What do you want to do now?
1) Report Bug
2) Debug
3) Run shell
4) Quit

Please make your choice from above:
---%<-snip->%---




Can this issue have a higher prioritized, please.
Why... Beaker automated testing with Fedora 35 is crippling
systems. Its leaving the bootorder wrong and marking systems broken in the double digits.
Alot of time is being spent by sysadmins scrambling to correct this issue :/

Best,
pbunyan

Comment 54 Jan Pazdziora 2022-05-25 14:11:19 UTC
I have no other info to give instead of agreeing that this needs to be fixed, even if the changes in the systemd package got reverted to the systemd-249.9-1.fc35 state (which is what comment 0 says was a working NVR).

Comment 55 PaulB 2022-05-25 14:21:28 UTC
(In reply to Jan Pazdziora from comment #54)
> I have no other info to give instead of agreeing that this needs to be
> fixed, even if the changes in the systemd package got reverted to the
> systemd-249.9-1.fc35 state (which is what comment 0 says was a working NVR).

Thank you for quick reply, Jan.

fwiw... Fedora36 installs work fine in Beaker.

[root@ampere-hr330a-06 ~]# uname -a
Linux ampere-hr330a-06.khw4.lab.eng.bos.redhat.com 5.17.9-300.fc36.aarch64 #1 SMP Wed May 18 14:50:13 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
[root@ampere-hr330a-06 ~]# cat /etc/redhat-release 
Fedora release 36 (Thirty Six)
[root@ampere-hr330a-06 ~]# yum list installed | grep systemd
rpm-plugin-systemd-inhibit.aarch64     4.17.0-10.fc36             @beaker-Fedora-Everything
systemd.aarch64                        250.3-8.fc36               @beaker-Fedora-Everything
systemd-libs.aarch64                   250.3-8.fc36               @beaker-Fedora-Everything
systemd-networkd.aarch64               250.3-8.fc36               @beaker-Fedora-Everything
systemd-oomd-defaults.noarch           250.3-8.fc36               @beaker-Fedora-Everything
systemd-pam.aarch64                    250.3-8.fc36               @beaker-Fedora-Everything
systemd-resolved.aarch64               250.3-8.fc36               @beaker-Fedora-Everything
systemd-udev.aarch64                   250.3-8.fc36               @beaker-Fedora-Everything

Comment 56 Arthur Benoit 2022-05-25 15:03:14 UTC
How do we address the fact that this is crippling aarch64 systems when beaker jobs are calling for Fedora 35? 
The kernel-hw queue had 40 or so broken systems that were started by CKI.

Comment 57 PaulB 2022-05-25 16:50:37 UTC
(In reply to PaulB from comment #55)
> (In reply to Jan Pazdziora from comment #54)
> > I have no other info to give instead of agreeing that this needs to be
> > fixed, even if the changes in the systemd package got reverted to the
> > systemd-249.9-1.fc35 state (which is what comment 0 says was a working NVR).
> 
> Thank you for quick reply, Jan.
> 
> fwiw... Fedora36 installs work fine in Beaker.
> 
> [root@ampere-hr330a-06 ~]# uname -a
> Linux ampere-hr330a-06.khw4.lab.eng.bos.redhat.com 5.17.9-300.fc36.aarch64
> #1 SMP Wed May 18 14:50:13 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
> [root@ampere-hr330a-06 ~]# cat /etc/redhat-release 
> Fedora release 36 (Thirty Six)
> [root@ampere-hr330a-06 ~]# yum list installed | grep systemd
> rpm-plugin-systemd-inhibit.aarch64     4.17.0-10.fc36            
> @beaker-Fedora-Everything
> systemd.aarch64                        250.3-8.fc36              
> @beaker-Fedora-Everything
> systemd-libs.aarch64                   250.3-8.fc36              
> @beaker-Fedora-Everything
> systemd-networkd.aarch64               250.3-8.fc36              
> @beaker-Fedora-Everything
> systemd-oomd-defaults.noarch           250.3-8.fc36              
> @beaker-Fedora-Everything
> systemd-pam.aarch64                    250.3-8.fc36              
> @beaker-Fedora-Everything
> systemd-resolved.aarch64               250.3-8.fc36              
> @beaker-Fedora-Everything
> systemd-udev.aarch64                   250.3-8.fc36              
> @beaker-Fedora-Everything



Rawhide works too:

[root@ampere-hr330a-06 ~]# uname -a
Linux ampere-hr330a-06.khw4.lab.eng.bos.redhat.com 5.18.0-60.fc37.aarch64 #1 SMP PREEMPT_DYNAMIC Mon May 23 13:52:56 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
[root@ampere-hr330a-06 ~]# date
Wed May 25 12:35:10 PM EDT 2022
[root@ampere-hr330a-06 ~]# cat /etc/redhat-release 
Fedora release 37 (Rawhide)
root@ampere-hr330a-06 ~]# yum list installed | grep systemd
rpm-plugin-systemd-inhibit.aarch64     4.18.0-0.alpha2.1.fc37     @beaker-Fedora-Everything
systemd.aarch64                        251-1.fc37                 @beaker-Fedora-Everything
systemd-libs.aarch64                   251-1.fc37                 @beaker-Fedora-Everything
systemd-networkd.aarch64               251-1.fc37                 @beaker-Fedora-Everything
systemd-oomd-defaults.noarch           251-1.fc37                 @beaker-Fedora-Everything
systemd-pam.aarch64                    251-1.fc37                 @beaker-Fedora-Everything
systemd-resolved.aarch64               251-1.fc37                 @beaker-Fedora-Everything
systemd-udev.aarch64                   251-1.fc37                 @beaker-Fedora-Everything



Issue seems to be isolated to aarch64 and F35 :/

Comment 58 Jan Pazdziora 2022-05-25 16:56:25 UTC
(In reply to PaulB from comment #57)
> 
> Issue seems to be isolated to aarch64 and F35 :/

It is isolated to Fedora 35 as mentioned in 5. However, it is not ARM specific, it affects all architectures.

Comment 59 Michael Hofmann 2022-05-31 08:55:33 UTC
(In reply to Arthur Benoit from comment #56)
> How do we address the fact that this is crippling aarch64 systems when
> beaker jobs are calling for Fedora 35? 
> The kernel-hw queue had 40 or so broken systems that were started by CKI.

To set the record straight, CKI works around this problem by running rawhide. So just for once we were not to blame for these particular broken machines 🙈.

Comment 61 Ken Teh 2022-06-01 11:48:45 UTC
I don't quite understand the discussion here but am I correct that the  bug still exists?  Ie, the everything netinst iso is still broken. I see comments about new builds being made available for testing.  I'd like to help test.  Can someone provide some pointers on how to go about downloading the "fixed" isos for testing?  Thanks.

Comment 62 Adam Williamson 2022-06-01 15:26:25 UTC
People keep asking when this is going to be "fixed". The problem is we've tried to fix it four times and it hasn't worked. Nobody has come up with a way to fully "fix" it in all cases yet. The problem is that there are various cases:

1. You run an install with all release-day packages (old anaconda, old systemd) then update systemd: should systemd's scriptlets try and do something to /etc/resolv.conf ?
2. You run an install with release-day anaconda but updated systemd (e.g. netinst install): what happens?
3. You run an install with updated anaconda and updated systemd (e.g. a live-respin or a manually generated image or something from openQA or Beaker): what happens?

Every approach we've tried so far, AIUI, has had problems in at least one of those cases, and nobody seems to have an idea for an approach that would solve them all.

This is different in Fedora 36 because we got anaconda and systemd lined up the way we want before release for Fedora 36, so it doesn't have the same considerations as 35 does.

Comment 63 Edgar Hoch 2022-06-01 16:51:43 UTC
I think that the first goal should be that the current stable release Fedora 35 should install with the Fedora provided images and repos without aborting the installation. This are your cases 1 and 2. If users was able to install Fedora 35 in the last few months, they should be able to do it until EOL. These users may already have a solution how to handle /etc/resolv.conf in this case.

In no case package installation should crash in it's scripts and make the system unusable or abort the (kickstart) installation.


(In reply to Adam Williamson from comment #62)
> 1. You run an install with all release-day packages (old anaconda, old
> systemd) then update systemd: should systemd's scriptlets try and do
> something to /etc/resolv.conf ?

I think it should do nothing with /etc/resolv.conf, because the system is already running in some way, possibly with a change by the user / administrator.

> 2. You run an install with release-day anaconda but updated systemd (e.g.
> netinst install): what happens?

"Same behavior" as before: It should leave the file as it would be with an old systemd.

> 3. You run an install with updated anaconda and updated systemd (e.g. a
> live-respin or a manually generated image or something from openQA or
> Beaker): what happens?

These people can use Fedora 36 where the problem is fixed, or use the old images? I would prioritize this lower, because these are not official Fedora images.


In my case, I have added the following script to the end (important!) of the %post part of my kickstart files for Fedora 35 because I know that NetworkManager and systemd-resolved are installed and will be running after reboot. It should be the last command running before reboot because it breaks dns name resolution. systemd package scripts cannot to this because they are not the last action that run on the installed system.

    if [[ ! -h /etc/resolv.conf ]] ; then
        ln -fsv ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
    fi

Comment 64 John Kissane 2022-06-02 11:03:29 UTC
I ran into this issue as well doing a kickstart install that I knew had worked in the past. My workaround before I found this bug report was to disable the local repository for the updates & install from base instead.

Comment 65 PaulB 2022-06-02 13:52:46 UTC
(In reply to Adam Williamson from comment #62)
> People keep asking when this is going to be "fixed". The problem is we've
> tried to fix it four times and it hasn't worked. Nobody has come up with a
> way to fully "fix" it in all cases yet. The problem is that there are
> various cases:
> 
> 1. You run an install with all release-day packages (old anaconda, old
> systemd) then update systemd: should systemd's scriptlets try and do
> something to /etc/resolv.conf ?
> 2. You run an install with release-day anaconda but updated systemd (e.g.
> netinst install): what happens?
> 3. You run an install with updated anaconda and updated systemd (e.g. a
> live-respin or a manually generated image or something from openQA or
> Beaker): what happens?
> 
> Every approach we've tried so far, AIUI, has had problems in at least one of
> those cases, and nobody seems to have an idea for an approach that would
> solve them all.
> 
> This is different in Fedora 36 because we got anaconda and systemd lined up
> the way we want before release for Fedora 36, so it doesn't have the same
> considerations as 35 does.

Adam,
The kernel-hw team provides and services a lions share of shared systems to Beaker for all users / groups to test with. This issue leaves all the systems cripples with broken boot order when they are tested with Fedora35. This takes alot of man hours to return the numerous broken systems back to working order, only for them to be set broken once again when a Fedora35 install is run.
Would it be acceptable for the kernel-hw team to disable Fedora35 on all their shared test hardware until this issue is resolved?

Best,
pbunyan

Comment 66 Jan Pazdziora 2022-06-02 14:04:15 UTC
(In reply to Adam Williamson from comment #62)
> People keep asking when this is going to be "fixed". The problem is we've
> tried to fix it four times and it hasn't worked.

With systemd-249.9-1.fc35, provisioning of Fedora 35 worked.

What were the issues with that systemd version-release and were they more severe than not being able to provision Fedora? Can't we just revert systemd package to that state, and have any fixes for other issues land in Fedora 36?

Comment 67 Adam Williamson 2022-06-02 15:43:50 UTC
Those are questions for the maintainers. I'm just the QA guy. I was trying to help people understand why this has not been resolved already.

Comment 68 Radek Vykydal 2022-06-03 08:20:11 UTC
*** Bug 2083411 has been marked as a duplicate of this bug. ***

Comment 69 Radek Vykydal 2022-06-03 08:21:25 UTC
*** Bug 2083866 has been marked as a duplicate of this bug. ***

Comment 70 Adam Williamson 2022-06-03 16:34:13 UTC
Zbigniew, can you chime in here? It doesn't seem like the current situation is sustainable. Is there still a way to make all three cases viable?

If we can only fix cases 1) and 2) in my list, but 3) can be made to work with an updates.img, I can live with that (I can have openQA always use the updates.img when building updated F35 images).

Comment 71 Adam Williamson 2022-06-10 22:47:51 UTC
OK, I think I see the problem here. Zbigniew's latest change didn't do what he thought it would. He added a check that makes the scriptlet only link /etc/resolv.conf to the dynamic stub if `/run/systemd/system/` is present; if it's not present, but /etc/resolv.conf doesn't exist, it gets linked to /usr/lib/systemd/resolv.conf . I think he expected the check would fail in the installer case, and we'd get a link to /usr/lib/systemd/resolv.conf .

That's not what happens, though. /run/systemd/system/ *does* exist in the installed system root in the installer environment. So the check passes, and we still create the dangling symlink whose target is in a non-existent directory, and anaconda still crashes.

I see two possible ways to fix this. One is to extend the check from:

if test -d /run/systemd/system/; then

to:

if test -d /run/systemd/system/ && test -d /run/systemd/resolve/; then

this would, I think, give the result Zbigniew intended in the installer environment: we'd get a symlink to ../usr/lib/systemd/resolv.conf , which should exist, which should cause anaconda not to crash (and not to replace it).

The other option is to extend the symlink creation from this:

ln -sv ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf || :

to this:

mkdir -p /run/systemd/resolve && touch /run/systemd/resolve/stub-resolv.conf && ln -sv ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf || :

i.e., create the symlink target if it is not there already. This would make the symlink's target an existent regular file, and make anaconda not attempt to replace it. Of course the target would be an empty file and name resolution wouldn't work inside the installed system root, but that shouldn't be a terrible problem. If we wanted to give it a shot at working, I guess, instead of touching the target, we could make the target itself a symlink to /usr/lib/systemd/resolv.conf , but that might be gilding the lily.

On regular boot of the installed system, the whole of /run should be recreated, and the symlink should wind up pointing to a stub that's actually there, and everything should work. i hope.

I'm going to test these two approaches now.

Comment 72 Adam Williamson 2022-06-11 00:06:19 UTC
So, I see that 沙包妖梦 actually suggested the "create /run/systemd/resolve/stub-resolv.conf" approach back in April in #c39, so credit to them for that (I came to the same idea independently).

I've tested this now. It does work. In response to Zbigniew's concerns from #c40, the installed system's `/run` *is* mounted during the install process (it is not just a regular directory). I ran a test install with the change I suggested above to create the directory and file when creating the symlink. anaconda does not crash, the symlink is successfully created, and anaconda leaves it alone. On booting the installed system, there are no warnings that I can see, and /etc/resolv.conf is correctly linked to ../run/systemd/resolve/stub-resolv.conf which exists and has correct contents. Name resolution works.

I also tested an install without systemd-resolved (I made a kickstart that excludes it). That install also succeeds. /etc/resolv.conf is created as a regular file 'generated by NetworkManager', and name resolution works.

I did find one very specific scenario where this proposal makes things worse: if you try to install systemd-resolved after installing without it.

If you start from the install without systemd-resolved and then install the current stable systemd-resolved, it replaces /etc/resolv.conf with a symlink to ../run/systemd/resolve/stub-resolv.conf . At first this does not work, because systemd-resolved is *enabled* but not *started* on install, and so the target of the symlink does not exist. If you start systemd-resolved.service, though, the target is created and /etc/resolv.conf becomes 'correct'.

If you start from the install without systemd-resolved and then install my patched systemd-resolved, it replaces /etc/resolv.conf with a symlink to ../run/systemd/resolve/stub-resolv.conf and *creates that as an empty file*, just like we told it to. If you then start systemd-resolved.service , it does not replace the empty stub file with a correct one, and so /etc/resolv.conf is still empty and doesn't work. You have to reboot before the correct stub file will exist.

That seems like a pretty minor issue, though, and probably not worth worrying about too hard? I don't think we need to worry about upgrade scenarios, since we switched to resolved-by-default in F33.

I'm gonna create an update for systemd with this change, but disable autopush. If Zbigniew has concerns, we can pull it.

Comment 73 Fedora Update System 2022-06-11 02:57:10 UTC
FEDORA-2022-6d2c62d6d6 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-6d2c62d6d6

Comment 74 Adam Williamson 2022-06-11 06:16:20 UTC
So in the update I went with the "create /run/systemd/resolve/stub-resolv.conf as a symlink to /usr/lib/systemd/resolv.conf if it doesn't exist yet" approach:

https://src.fedoraproject.org/rpms/systemd/c/6b234deaf8576d78278cd5d8aa8d77011ee1f12b?branch=f35

please test it out and see how it goes. Thanks.

Comment 75 Fedora Update System 2022-06-12 01:18:55 UTC
FEDORA-2022-6d2c62d6d6 has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-6d2c62d6d6`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-6d2c62d6d6

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 76 Edgar Hoch 2022-06-12 13:04:59 UTC
I have tested systemd-249.12-4.fc35.

With kickstart installation, it doesn't work as expected. DNS search domain only contains "." (as of file /usr/lib/systemd/resolv.conf) instead of our own dns search list which was provided by dhcp.

I don't find a proof in /var/log/anaconda/*, but I think that Anaconda or NetworkManager is creating the system-connection files in /etc/NetworkManager/system-connections/*.connection after package systemd-resolved has been installed, and after the scripts of package systemd-resolved has been run, which set the dns search list to ".", overwriting the value that the system got by dhcp.

It works as expected with the original Fedora 35 iso images without updates during kickstart installation.

The problem is that after reboot unqualified (short) host names are no longer resolved.


Update systemd* on a already working Fedora 35 installation works (because it doesn't change resolv.conf).

Comment 77 Adam Williamson 2022-06-12 14:52:47 UTC
Hmm. OK, I'll look into that tomorrow. I can go back to a build which creates the symlink target as an empty file and see if it behaves any differently in that case...

Comment 78 Adam Williamson 2022-06-12 15:58:22 UTC
Actually, I'm on vacation tomorrow and Tuesday, will look Wednesday if nobody else does. It should be quite easy to tweak the build to create the symlink target as an empty file if you want to try it that way.

Comment 79 Edgar Hoch 2022-06-13 21:04:58 UTC
I have build systemd with "%posttrans resolved" creating an empty file /run/systemd/resolve/stub-resolv.conf . This does not work either, the dns search list is empty after installation as of systemd-249.12-4.fc35.

I have build systemd with "%posttrans resolved" just creating /run/systemd/resolve, but not creating files in it and not creating a symlink (and printing some debug info for me). This works for me, because it leaves /etc/resolv.conf in the state created by NetworkManager (containing a "search" and three "nameserver" lines). I use a kickstart "%post" script that removes /etc/resolv.conf and creates the symlink to /run/systemd/resolve/stub-resolv.conf as the very last action of the "%post" script.

I think the general solution was done in Anaconda for Fedora 36.

For Fedora 35 we need something that is run just before reboot, but I'm unsure if this is possible (beside of administrator kickstart %post script). So leaving /etc/resolv.conf as a plain file is not a perfect solution, but it is the same state as installation from Fedora 35 release images without repo updates and it doesn't abort installation.

Comment 80 Adam Williamson 2022-06-15 01:29:10 UTC
The thing that was changed in anaconda for F36 is that it now doesn't replace /etc/resolv.conf if it's a dangling symlink . So systemd-resolved %post can create it as a dangling symlink and anaconda leaves it that way. This was changed in F35 anaconda updates too, but of course GA images still have the original behaviour.

Your "This works for me, because it leaves /etc/resolv.conf in the state created by NetworkManager" would be just the same as reverting systemd to exactly the state it was in at F35 release, because that's what happens with that setup - you get /etc/resolv.conf controlled by NM. This is what we started out trying to *fix*, because it's not what was intended.

There's no way to lever in something that happens "just before reboot" for the scenario we're trying to handle here (netinst install with GA images), because to do that we have to change anaconda's behaviour, and we can't change anaconda's behaviour on the GA images. That's fixed. We can only make changes in the *packages that the netinst install deploys*, and the latest they can do anything is in %posttrans, which is already where the systemd-resolved scriptlet we're poking runs.

Comment 81 Edgar Hoch 2022-06-15 01:58:35 UTC
Thanks for the explanation.

I think another change between F35 and F36 is that in F36 systemd-resolved is running during installation (I run "ps -ef" during kickstart %post to see what is running), while on F35 it isn't running.

Maybe a solution would be to start systemd-resolved.service (only during system installation!) and wait for finishing startup, and only then set up the symlink to already existing files /run/systemd/resolve/stub-resolv.conf and /run/systemd/resolve/resolv.conf? Then dns resolution would still work during installation, and (hopefully) NetworkManager can save the right values of "dns-search" in /etc/NetworkManager/system-connections/? I don't know if it can work, it's just a guess.

Comment 82 Edgar Hoch 2022-06-15 19:50:44 UTC
I will try another build which starts systemd-resolved. What do you think about this?

%posttrans resolved
test -e %{_localstatedir}/lib/rpm-state/systemd-resolved-initial-installation || exit 0
# Initial installation
rm %{_localstatedir}/lib/rpm-state/systemd-resolved-initial-installation || :
### ... comments not listed here

if systemctl -q is-enabled systemd-resolved.service &>/dev/null &&
   ! systemd-analyze cat-config systemd/resolved.conf 2>/dev/null |
        grep -iqE '^DNSStubListener\s*=\s*(no?|false|0|off)\s*$' &&
   ! mountpoint /etc/resolv.conf &>/dev/null; then

  if ! systemctl is-active systemd-resolved ; then
    systemctl start systemd-resolved || :
  fi
  if test -d /run/systemd/resolve/ &&
    ! mountpoint /etc/resolv.conf &>/dev/null; then
    ln -fsv ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf &>/dev/null || :
  fi
fi

Comment 83 Adam Williamson 2022-06-15 20:01:45 UTC
I suspect it might not work because I don't think systemd will start units inside a chroot.

Also note that "initial installation" (as the spec file checks for) is not strictly the same thing as "system installation". "Initial installation" can also be "first time systemd-resolved is installed on a system that has already been deployed". It would also be encountered during things like live image builds, which are run through anaconda.

Comment 84 Edgar Hoch 2022-06-15 20:44:41 UTC
Yes, you are right. systemd-resolved wasn't started. /var/log/anaconda/packages.log contains
Running in chroot, ignoring command 'is-active'

I guess the last attempt would be to create an updates.img, because systemd-resolved should run outside of the chroot?

Comment 85 Adam Williamson 2022-06-15 20:56:28 UTC
Well, it'd be trivial (I think) to ship an updates.img that solves the problem - just backport the anaconda changes from F36. But that's not going to make everyone happy, is it? Folks inside RH are unhappy because they want their beaker stuff to work, and "fixing" it with an updates image requires everyone to update their config to use the updates image. Any random person just trying to do an F35 netinst is not going to know they need an updates image, they'll try it, it'll fail, then *hopefully* they'll find this bug and try again with the updates image. Which isn't a great experience. I'd much rather we ship an update that makes installs not crash.

If we can't fix the problem you noted in my update, we may just have to go with the idea of reverting to the GA behavior, i.e., let the NM-controlled /etc/resolv.conf be copied over from the install environment. That means choosing not to fix #2032085 for future F35 network installs and living with whatever problems/inconsistencies that causes.

Comment 86 Adam Williamson 2022-06-15 22:06:24 UTC
Edgar: so, in testing, I can't seem to reproduce your result.

I'm now testing on a bare metal test box on my home network. I configured my router to set a search domain for DHCP purposes. I checked that if I do an F36 network install, the line 'search happyassassin.net' (the domain I set) appears in /etc/resolv.conf and /run/systemd/resolve/resolv.conf .

I then did an F35 network install from the GA netinst image, with a repo containing my systemd package added as an additional repo. If I check /etc/resolv.conf in the installed system root *before rebooting*, it does not have the 'search happyassassin.net' line (as I'd expect, given how my attempted fix works). /run/systemd/resolve/resolv.conf does not exist in the installed system root at all yet, again as expected.

But if I then reboot to the installed system and check again, all is good. /run/systemd/resolve/stub-resolv.conf is now a regular file not a symlink to /usr/lib/systemd/resolv.conf , and it has the 'search happyassassin.net' line. /etc/resolv.conf is a symlink to /run/systemd/resolve/stub-resolv.conf and of course is the same. /run/systemd/resolve/resolv.conf exists and has the 'search happyassassin.net' line.

So, AFAICS, all is working as it should once I reboot to the installed system. Could you provide any additional details on the test scenario which didn't work as expected for you?

Comment 87 Edgar Hoch 2022-06-15 22:16:57 UTC
I assume you are using dynamic network configuration after reboot?

I need to mention that I configure kickstart to create a static network configuration for our workstations and servers, because these should not depend on a running DHCP server.
The DHCP server is running for notebooks and for (all) system installation with PXE.

network --bootproto=static --device=bootif --ip=XXX --netmask=255.255.255.0 --gateway=YYY --noipv6 --nameserver=ZZZ --hostname=NNN

I havn't tested it, but all systems that uses (only) dynamic network configuration are not affected by the dns search domain problem, because they will receive actual data at each boot (more precisely: every time a network connection is started).

So, our configuration may be a unusual case?

Comment 88 Adam Williamson 2022-06-16 01:33:32 UTC
Yeah, that's actually closer to what I first thought you were saying - that you get a problem with a fairly specific scenario with kickstart network configuration. So your expectation is that you configure *static* networking via kickstart, but that the DNS configuration (or just the search domains?) are retrieved via DHCP all the same? I am not a networking expert so I confess I don't know if that's expected behaviour. You say this is how things work with F35 GA (including GA systemd) though, right? Does it also work as you expect with F36 GA?

Either way, it does make me think it might be OK to just send my update stable, if it will at least solve the crashes and make things work as intended in more common cases.

Comment 89 Edgar Hoch 2022-06-16 02:08:33 UTC
As far as I know there is no kickstart option to configure the dns search domain. So I cannot configure it this way.

For system installation I usually use PXE boot, which gets all network configuration from DHCP, including dns server, dns search domain, time servers, bootp server, etc. It boots into pxelinux (for legacy bios) or grub2 (for uefi systems) and presents a list of boot options (which I have described in the corresponding config files). This allows booting a linux kernel and pass to it kernel boot parameters (initrd, inst.repo, inst.ks, etc.). It searches for a kickstart file on our nfs server and starts the installation.

The installation system get all network configuration from DHCP. The kickstart configuration causes that the configured values and other current parameters are stored on the installed system for use after reboot.

Yes, dns search domain comes only from DHCP, and it works with Fedora 36, and it has worked the same way on every previous Fedora version ever (as far as I remember, it's so many years). Only this new systemd updates on Fedora 35 has broken this installation method.

Yes, it is ok to send your update to stable, because it makes the current situation better, as I also wrote in bodhi.


If others are lucky with this solution, then I can create a workaround for our purpose, and we can save the time to search for further (unsatisfactory) solutions, because it works again with Fedora 36.

Many thanks for your time!

Comment 90 Jan Pazdziora 2022-06-16 04:00:08 UTC
Adam, note that installation with systemd-249.12-4.fc35.x86_64 fails on beaker because in the Running post-installation scripts, the fetch / curl operations cannot resolve the lab controller's hostname.

Comment 93 Adam Williamson 2022-06-16 15:59:33 UTC
For non-RHers who notice a couple of private comments above - they're about an internal RH system, but they point up a general issue with my update: kickstart scripts may run in the installed system root and expect name resolution to work. With F35 GA this worked, with the proposed fix it does not. I'll see if there's anything I can come up with to change this.

Comment 94 Adam Williamson 2022-06-16 18:07:07 UTC
OK, I think I reproduced the problem with %post scripts that need to access the network, and fixed it without (hopefully) breaking anything else. I've put a -5 build in the update. Jan, can you please try that? Edgar, can you also see if it fixes your similar issue with %post and NFS? I do not expect it to fix your DHCP search domain issue, but if I'm understanding that correctly I do think it'd be reasonable to push this out and think about that later.

Thanks!

Comment 95 Adam Williamson 2022-06-16 18:10:17 UTC
Sorry, forgot to mention - what the -5 build actually does is change the target of the /run/systemd/resolve/stub-resolv.conf symlink we create during initial package install. Instead of pointing to /usr/lib/systemd/resolv.conf , which is a static stub file that always has the same contents and says to use 127.0.0.53 , it now points to resolv.conf in the same directory, which should point to the "real" nameservers.

This is still just a temporary setup that only exists until reboot (when /run will be regenerated and /run/systemd/resolve/stub-resolv.conf will be the proper dynamic stub file, not a symlink to somewhere else).

Comment 96 Fedora Update System 2022-06-17 01:48:55 UTC
FEDORA-2022-6d2c62d6d6 has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-6d2c62d6d6`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-6d2c62d6d6

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 97 Edgar Hoch 2022-06-17 05:21:50 UTC
Adam, this new update fixes my problems with %post and NFS. It works as usual now, and after reboot my own postinstallation scripts are automatically started, and they can also use NFS.

DNS search domain isn't preserved, but it doesn't matter, dns resolution works now, and my scripts already used fully qualified domain names. Also, my scripts has host class specific dns search domain lists, and they set the dns search domain if it doesn't match the value from our preset list. It works for me, you don't need to search for solution about this problem because of me.

Thanks for your work!

Comment 98 Jan Pazdziora 2022-06-17 13:27:50 UTC
I also confirm that with systemd-249.12-5.fc35, things are sane again. Karma in bodhi given.

Thank you!

Comment 100 Adam Williamson 2022-06-18 01:43:10 UTC
Zbigniew, can you please check my fix? I don't want to push it stable without your signoff, but since it does at least prevent installs crashing, if I don't hear either way by next week, I'll push it.

Comment 101 Fedora Update System 2022-06-21 01:08:18 UTC
FEDORA-2022-6d2c62d6d6 has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 102 Zbigniew Jędrzejewski-Szmek 2022-06-22 15:21:04 UTC
AdamW, thanks for working on this and apologies for not getting to this earlier.
Since it seems to work, then I'm all for it :)

Comment 103 Adam Williamson 2022-06-22 15:58:54 UTC
Thanks Zbigniew! Since nobody complained I went ahead and pushed it stable yesterday, so we should be good.

Comment 104 John Villalovos 2022-06-22 16:41:45 UTC
Thanks for this. I am now getting a similar error when doing an install

    Performing post-installation setup tasks
    Configuring installed system
    Writing network configuration
    An unknown error has occured, look at the /tmp/anaconda-tb* file(s) for more details  

I press '3' on the console and then get:


    return self._get_method_reply(
  File "/usr/lib64/python3.10/site-packages/pyanaconda/modules/common/task/__init__.py", line 46, in sync_run_task
    task_proxy.Finish()
  File "/usr/lib64/python3.10/site-packages/pyanaconda/network.py", line 293, in write_configuration
    sync_run_task(task_proxy)
  File "/usr/lib64/python3.10/site-packages/pyanaconda/installation_tasks.py", line 458, in run_task
    self._task(*self._task_args, **self._task_kwargs)
  File "/usr/lib64/python3.10/site-packages/pyanaconda/installation_tasks.py", line 496, in start
    self.run_task()
  File "/usr/lib64/python3.10/site-packages/pyanaconda/installation_tasks.py", line 311, in start
    item.start()
  File "/usr/lib64/python3.10/site-packages/pyanaconda/installation_tasks.py", line 311, in start
    item.start()
  File "/usr/lib64/python3.10/site-packages/pyanaconda/installation_tasks.py", line 311, in start
    item.start()


  File "/usr/lib64/python3.10/site-packages/pyanaconda/installation.py", line 415, in run_installation
    queue.start()
  File "/usr/lib64/python3.10/threading.py", line 946, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib64/python3.10/site-packages/pyanaconda/threading.py", line 275, in run
    threading.Thread.run(self)
dasbus.error.DBusError: [Errno 2] No such file or directory: '/mnt/sysroot/etc/resolv.conf'


This is different in that before the error message about resolv.conf was displayed on the screen during the install. I didn't need to press '3'.

I checked our mirror repository log and it downloaded systemd-249.12-3.fc35.aarch64.rpm

Comment 105 John Villalovos 2022-06-22 16:43:34 UTC
I guess I may be wrong that it is different. As I see the initial bug said they also had to check the log file. So I might be mis-remembering that it showed up in the console log.

Comment 106 Adam Williamson 2022-06-22 17:04:32 UTC
"I checked our mirror repository log and it downloaded systemd-249.12-3.fc35.aarch64.rpm"

The fix is in 249.12-5.fc35. It looks like your mirror isn't up to date yet.

Comment 107 Ken Teh 2022-06-22 17:26:45 UTC
I need some help on how to use this update together with the current netinst iso that is available from the mirrors. Do I need to rebuild the iso? Or, is there a magic command to swap in this update during the install?  Thanks.

Comment 108 John Villalovos 2022-06-22 17:30:19 UTC
(In reply to Adam Williamson from comment #106)
> "I checked our mirror repository log and it downloaded
> systemd-249.12-3.fc35.aarch64.rpm"
> 
> The fix is in 249.12-5.fc35. It looks like your mirror isn't up to date yet.

Thanks! We mirror from `mirrors.kernel.org` and looks like they haven't yet synced. :(

Digging deeper into it it doesn't seem like they have anything newer than 1-June-2022 in the `s` updates directory.

https://da.mirrors.kernel.org/fedora/updates/35/Everything/aarch64/Packages/s/

Comment 109 Adam Williamson 2022-06-22 19:17:28 UTC
Ken: you don't need to do anything special. As long as the install hits on a mirror that has got the update, things should work without you having to do anything special.

John: hmm, that seems like an issue. Thanks for spotting it. I'll try and get someone to follow up (I see you're in IRC poking about this too).

Comment 110 John Villalovos 2022-06-22 21:40:06 UTC
This fix is working for us on aarch64 architecture. Thanks!


Note You need to log in before you can comment on or make changes to this bug.