Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1861463 - zram-setup: Failed to load configuration: No such file or directory
Summary: zram-setup@zram0.service: Failed to load configuration: No such file or direc...
Keywords:
Status: NEW
Alias: None
Product: Fedora
Classification: Fedora
Component: udisks2
Version: 33
Hardware: Unspecified
OS: Linux
unspecified
unspecified
Target Milestone: ---
Assignee: systemd-maint
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: ARMTracker
TreeView+ depends on / blocked
 
Reported: 2020-07-28 17:09 UTC by Paul Whalen
Modified: 2020-08-24 18:52 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug
Embargoed:


Attachments (Terms of Use)
zram systemd debug log (7.15 KB, text/plain)
2020-07-28 17:11 UTC, Paul Whalen
no flags Details
Full boot log (840.84 KB, text/plain)
2020-07-28 17:12 UTC, Paul Whalen
no flags Details
Full boot logs on aarch64 (Fedora-Rawhide-20200811.n.0) (3.55 MB, text/plain)
2020-08-12 18:45 UTC, Paul Whalen
no flags Details

Description Paul Whalen 2020-07-28 17:09:48 UTC
Description of problem:
When attempting to install an armhfp system, the boot hangs just before the installer with no indication why, with systemd debug enabled:

zram-setup: Failed to load configuration: No such file or directory
zram-setup: Failed to load configuration: No such file or directory
zram-setup: Failed to load configuration: No such file or directory


Full logs attached. 


Version-Release number of selected component (if applicable):
systemd-246~rc2-2.fc33

How reproducible:
Fails on hardware, working in qemu.

Comment 1 Paul Whalen 2020-07-28 17:11:14 UTC
Created attachment 1702695 [details]
zram systemd debug log

Comment 2 Paul Whalen 2020-07-28 17:12:28 UTC
Created attachment 1702696 [details]
Full boot log

Comment 3 Paul Whalen 2020-07-28 17:14:39 UTC
cc'ing cmurf

Comment 4 Chris Murphy 2020-07-28 19:20:10 UTC
>dev-zram0.mount: Failed to load configuration: No such file or directory

I don't have a mount unit for this on any working setup. But maybe this is just a distraction. I'm not sure what it failed to load.

Does /usr/lib/systemd/zram-generator.conf exist on this media? And is the single uncommented line merely [zram0] ?

Comment 5 Chris Murphy 2020-07-28 19:30:33 UTC
The module loads super late...

[  174.399815] zram: Added device: zram0 succeeded.


Missing from the journal is something like:

[   34.258490] fedora kernel: zram0: detected capacity change from 0 to 1515192320

Comment 6 Paul Whalen 2020-07-28 20:15:25 UTC
(In reply to Chris Murphy from comment #4)
> >dev-zram0.mount: Failed to load configuration: No such file or directory
> 
> I don't have a mount unit for this on any working setup. But maybe this is
> just a distraction. I'm not sure what it failed to load.
> 
> Does /usr/lib/systemd/zram-generator.conf exist on this media? And is the
> single uncommented line merely [zram0] ?

This was a pxe installation attempt, the squashfs does include zram-generator.conf with '[zram0]' uncommented.

Comment 7 Zbigniew Jędrzejewski-Szmek 2020-08-01 12:20:56 UTC
zram-setup@.service is not part of the zram-generator package. It seems something is pulling in
this unit. I'm not sure what is the source is.

Comment 8 Zbigniew Jędrzejewski-Szmek 2020-08-01 13:20:53 UTC
My proposal:
1. add Obsoletes: udisk2-zram in zram-generator-defaults
2. drop udisk2-zram

Comment 9 Chris Murphy 2020-08-01 14:52:36 UTC
Sorry I missed this. I looked again at comps and kickstarts, and can't find any reference to udisks2-zram. I'm not sure why it's present in this case. Is there something in the 'arm-tools' group that might be pulling it in?

On the one hand, since it's not installed by default anywhere else, I'm not sure how useful it is. On the other hand, I could see useful integration on server+cloud+desktop via Cockpit and Disk Utility, to (a) change the zram-based swap configuration (b) create generic non-swap ram disks. Perhaps in the short term all udisks2 needs is to not conflict with zram-generator by checking if there's already zram-based swap, and if so, then don't try to create another one. I think that might be all that's going on here, is zram-generator simply gets there first, and then anything else that assumes it can create a zram0 swap device fails. Noisy but non-fatal.

Comment 10 Tomáš Bžatek 2020-08-03 09:17:46 UTC
I wonder why this even fails the boot process...

We can't drop udisks2-zram as it provides management API that others may use. Short-term we can add Conflicts: zram-generator into udisks2-zram and long-term we can port udisks2-zram to use zram-generator as a backend, keeping the public API.

Comment 11 Zbigniew Jędrzejewski-Szmek 2020-08-03 10:21:00 UTC
(In reply to Tomáš Bžatek from comment #10)
> I wonder why this even fails the boot process...

I'm not sure if this is actually the cause of the problems. Those messages
zram-setup: Failed to load configuration: No such file or directory
would only be fatal if zram-setup was a dependency of something important.
Is the unit configured to be Required or PartOf something?

> We can't drop udisks2-zram as it provides management API that others may
> use. Short-term we can add Conflicts: zram-generator into udisks2-zram and

If anything, udisks2-zram could conflict with zram-generator-defaults. But
conflicts can be problematic for users if those two units are both pulled in as
dependencies. So I think it'd be better to make the two services aware of each other.
Is there any simple check I could add in zram-generator to disable itself if
udisk2-zram is going to be active?

> long-term we can port udisks2-zram to use zram-generator as a backend,
> keeping the public API.

I'm not sure if this is worth the trouble. The actual process of creating a zram
device is so simple. Instead, if both implementations have configurations,
one of them should disable itself with a nice message.

Comment 12 Chris Murphy 2020-08-03 17:14:36 UTC
I'm not sure what udisks2-zram even does. I thought arm installs were originally depending on 'zram' package to do this, not 'udisks2-zram'.

zram-generator is likely to beat all other implementations, it just runs that much earlier in the startup process.

Comment 13 Tomáš Bžatek 2020-08-04 15:08:44 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #11)
> I'm not sure if this is actually the cause of the problems. Those messages
> zram-setup: Failed to load configuration: No such file or
> directory
> would only be fatal if zram-setup was a dependency of
> something important.
> Is the unit configured to be Required or PartOf something?

Nothing I'm aware of. https://raw.githubusercontent.com/storaged-project/udisks/master/modules/zram/data/zram-setup%40.service

Apparently Anaconda has been shipping their own zram.service: https://github.com/rhinstaller/anaconda/pull/2727 (thanks to Vojtech Trefny for noticing that).

> If anything, udisks2-zram could conflict with zram-generator-defaults. But
> conflicts can be problematic for users if those two units are both pulled in
> as
> dependencies. So I think it'd be better to make the two services aware of
> each other.
> Is there any simple check I could add in zram-generator to disable itself if
> udisk2-zram is going to be active?

The udisks2-zram module creates a file "$(libdir)/zram.conf.d/zramN" for each configured zram device where N is an index. See related EnvironmentFile key in the mentioned service file.
.... aaand looks like we found a bug. The udisks2-zram code is creating config files in /usr/lib64/zram.conf.d on x86_64 yet the service file hardcodes /usr/local/lib/zram.conf.d. Clearly nobody is using that code in reality.

Besides that it creates /usr/lib/modules-load.d/zram.conf and /usr/lib/modprobe.d/zram.conf files (with no indication of being created by UDisks2).

> I'm not sure if this is worth the trouble. The actual process of creating a
> zram
> device is so simple. Instead, if both implementations have configurations,
> one of them should disable itself with a nice message.

Right. There should at least be some degree of awareness since the udisks2-zram module exposes an extra D-Bus interface for management over any existing /dev/zram* block device even when created by a different framework.
Filed upstream: https://github.com/storaged-project/udisks/issues/791

Comment 14 Ben Cotton 2020-08-11 13:50:27 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 33 development cycle.
Changing version to 33.

Comment 15 Paul Whalen 2020-08-12 18:42:59 UTC
This continues to break installs on arm and aarch64 SBC's. Proposing as a blocker for Fedora 33 Beta, criteria: "The installer must run when launched normally from the release-blocking images."

Comment 16 Paul Whalen 2020-08-12 18:45:15 UTC
Created attachment 1711215 [details]
Full boot logs on aarch64 (Fedora-Rawhide-20200811.n.0)

Comment 17 Tomáš Bžatek 2020-08-13 16:06:39 UTC
I've pushed udisks2-2.9.1-1.fc33 (bodhi FEDORA-2020-e143a283fb) that brings couple of changes outlined above but probably nothing major that would help to fix this issue. The only notable change is a renamed udisks2-zram-setup@.service unit to help distinguish service origin. I haven't added any Conflicts: for the moment. The rest of the udisks2-zram issues are tracked upstream in the above mentioned ticket.

I've check the comps and the Cockpit spec file, no traces of udisks2-zram Requires:. As far as I know no other package should explicitly require udisks2-zram. Kicking that package from installer image and default install should fix this bugzilla, however the reason it's pulled in needs to be found first. Please reassign as appropriate, this doesn't seem to be a udisks2 bug.

Comment 18 Chris Murphy 2020-08-17 07:06:47 UTC
I don't see udisks2-zram package being brought in on either aarch64 or armv7hl image creation. These are from 20200816

https://kojipkgs.fedoraproject.org//work/tasks/9245/49389245/oz-aarch64.log
https://kojipkgs.fedoraproject.org//work/tasks/9171/49389171/mock_output.log

I also only find expected items in the root and initramfs on both
Fedora-Workstation-33-20200816.n.0.aarch64.raw.xz
Fedora-Workstation-armhfp-33-20200816.n.0-sda.raw.xz

Comment 19 Tomáš Bžatek 2020-08-17 09:23:55 UTC
So does that mean the original issue is fixed? I don't see anything fishy in the logs either.

Comment 20 Chris Murphy 2020-08-17 13:36:03 UTC
I'm not sure, I don't have hardware. In the description it says the problem happens on hardware, not in qemu. That is also strange.

Comment 21 Paul Whalen 2020-08-17 17:11:39 UTC
Testing the compose from Fedora-33-20200816.n.0, I am no longer hitting this on arm or aarch64, installs are completed as expected.

Comment 22 Kamil Páral 2020-08-20 12:58:43 UTC
Based on comment 21, I'm unproposing this as a Beta blocker, the issue has disappeared. Paul, if it happens again, please nominate this bug again, thanks!

Comment 23 Zbigniew Jędrzejewski-Szmek 2020-08-21 05:44:30 UTC
> I'm not sure, I don't have hardware. In the description it says the problem happens on hardware, not in qemu. That is also strange.

I think this must be a false conclusion: there is nothing in either the udev rules or the
unit installed by udisks2-zram that is hardware specific. Most likely there is some race
condition and we just don't happen to see it on qemu.

I installed udisks2-zram on my rpi3, and what happens now is that both zram-generator and udisk2-zram
try to configure the same zram0 device:
Aug 21 01:30:49 rpi3 sudo[2202]:  zbyszek : TTY=pts/0 ; PWD=/home/zbyszek ; USER=root ; COMMAND=/usr/sbin/modprobe -v zram
Aug 21 01:30:49 rpi3 kernel: zram: Added device: zram0
Aug 21 01:30:49 rpi3 systemd[1]: Starting Create swap on /dev/zram0...                    <-------------- this is zram-generator
Aug 21 01:30:49 rpi3 systemd[1]: Starting UDisks2 managed ZRAM device zram0 setup...      <-------------- this is udisk2-zram
Aug 21 01:30:49 rpi3 kernel: zram0: detected capacity change from 0 to 489684992
Aug 21 01:30:49 rpi3 systemd[1]: udisks2-zram-setup: Succeeded.
Aug 21 01:30:49 rpi3 systemd[1]: Finished UDisks2 managed ZRAM device zram0 setup.
Aug 21 01:30:49 rpi3 zram-generator[2219]: Setting up swapspace version 1, size = 467 MiB (489680896 bytes)
Aug 21 01:30:49 rpi3 zram-generator[2219]: no label, UUID=1545938f-92d5-477e-a02d-0e8d5856a56e
Aug 21 01:30:49 rpi3 systemd[1]: Finished Create swap on /dev/zram0.
Aug 21 01:30:50 rpi3 systemd[1]: Activating swap Compressed swap on /dev/zram0...
Aug 21 01:30:50 rpi3 kernel: Adding 478204k swap on /dev/zram0.  Priority:100 extents:1 across:478204k SSFS
Aug 21 01:30:50 rpi3 systemd[1]: Activated swap Compressed swap on /dev/zram0.

The order of messages should not be trusted. They two things are executing in parallel and
happen "at the same time", so we cannot say which one succeeded. udisks2-zram-setup@.service
ignores all errors, while swap-create@.service doesn't, so I think the latter must be winning
the race here, but it's possible that they are both partially active. The kernel allows settings
to be set multiple times before the device is activated.

udisks2-zram-setup@.service will always execute because of the udev rule in
/usr/lib/udev/rules.d/90-udisks2-zram.rules. I don't think we want to have both active,
this can only lead to problems. Should I add a rule to zram-generator to disable itself
if /usr/lib/udev/rules.d/90-udisks2-zram.rules exists? This is ugly, but probably better
than status quo.

Comment 24 Chris Murphy 2020-08-21 17:08:05 UTC
>Should I add a rule to zram-generator to disable itself if /usr/lib/udev/rules.d/90-udisks2-zram.rules exists? This is ugly, but probably better than status quo.

I have no strong opinion. The conflict seems unlikely. I don't see udisks2-zram on either armv7hl or aarch64 media, and if it does pop up somewhere on media, that's a bug that definitely needs fixed. But to avoid a problem if someone intentionally installs udisks2-zram? Possibly.


Note You need to log in before you can comment on or make changes to this bug.