Bug 1861463

Created attachment 1702695 [details]
zram systemd debug log

Created attachment 1702696 [details]
Full boot log

cc'ing cmurf

>dev-zram0.mount: Failed to load configuration: No such file or directory

I don't have a mount unit for this on any working setup. But maybe this is just a distraction. I'm not sure what it failed to load.

Does /usr/lib/systemd/zram-generator.conf exist on this media? And is the single uncommented line merely [zram0] ?

The module loads super late...

[  174.399815] zram: Added device: zram0 succeeded.


Missing from the journal is something like:

[   34.258490] fedora kernel: zram0: detected capacity change from 0 to 1515192320

(In reply to Chris Murphy from comment #4)
> >dev-zram0.mount: Failed to load configuration: No such file or directory
> 
> I don't have a mount unit for this on any working setup. But maybe this is
> just a distraction. I'm not sure what it failed to load.
> 
> Does /usr/lib/systemd/zram-generator.conf exist on this media? And is the
> single uncommented line merely [zram0] ?

This was a pxe installation attempt, the squashfs does include zram-generator.conf with '[zram0]' uncommented.

zram-setup@.service is not part of the zram-generator package. It seems something is pulling in
this unit. I'm not sure what is the source is.

My proposal:
1. add Obsoletes: udisk2-zram in zram-generator-defaults
2. drop udisk2-zram

Sorry I missed this. I looked again at comps and kickstarts, and can't find any reference to udisks2-zram. I'm not sure why it's present in this case. Is there something in the 'arm-tools' group that might be pulling it in?

On the one hand, since it's not installed by default anywhere else, I'm not sure how useful it is. On the other hand, I could see useful integration on server+cloud+desktop via Cockpit and Disk Utility, to (a) change the zram-based swap configuration (b) create generic non-swap ram disks. Perhaps in the short term all udisks2 needs is to not conflict with zram-generator by checking if there's already zram-based swap, and if so, then don't try to create another one. I think that might be all that's going on here, is zram-generator simply gets there first, and then anything else that assumes it can create a zram0 swap device fails. Noisy but non-fatal.

I wonder why this even fails the boot process...

We can't drop udisks2-zram as it provides management API that others may use. Short-term we can add Conflicts: zram-generator into udisks2-zram and long-term we can port udisks2-zram to use zram-generator as a backend, keeping the public API.

(In reply to Tomáš Bžatek from comment #10)
> I wonder why this even fails the boot process...

I'm not sure if this is actually the cause of the problems. Those messages
zram-setup: Failed to load configuration: No such file or directory
would only be fatal if zram-setup was a dependency of something important.
Is the unit configured to be Required or PartOf something?

> We can't drop udisks2-zram as it provides management API that others may
> use. Short-term we can add Conflicts: zram-generator into udisks2-zram and

If anything, udisks2-zram could conflict with zram-generator-defaults. But
conflicts can be problematic for users if those two units are both pulled in as
dependencies. So I think it'd be better to make the two services aware of each other.
Is there any simple check I could add in zram-generator to disable itself if
udisk2-zram is going to be active?

> long-term we can port udisks2-zram to use zram-generator as a backend,
> keeping the public API.

I'm not sure if this is worth the trouble. The actual process of creating a zram
device is so simple. Instead, if both implementations have configurations,
one of them should disable itself with a nice message.

I'm not sure what udisks2-zram even does. I thought arm installs were originally depending on 'zram' package to do this, not 'udisks2-zram'.

zram-generator is likely to beat all other implementations, it just runs that much earlier in the startup process.

(In reply to Zbigniew Jędrzejewski-Szmek from comment #11)
> I'm not sure if this is actually the cause of the problems. Those messages
> zram-setup: Failed to load configuration: No such file or
> directory
> would only be fatal if zram-setup was a dependency of
> something important.
> Is the unit configured to be Required or PartOf something?

Nothing I'm aware of. https://raw.githubusercontent.com/storaged-project/udisks/master/modules/zram/data/zram-setup%40.service

Apparently Anaconda has been shipping their own zram.service: https://github.com/rhinstaller/anaconda/pull/2727 (thanks to Vojtech Trefny for noticing that).

> If anything, udisks2-zram could conflict with zram-generator-defaults. But
> conflicts can be problematic for users if those two units are both pulled in
> as
> dependencies. So I think it'd be better to make the two services aware of
> each other.
> Is there any simple check I could add in zram-generator to disable itself if
> udisk2-zram is going to be active?

The udisks2-zram module creates a file "$(libdir)/zram.conf.d/zramN" for each configured zram device where N is an index. See related EnvironmentFile key in the mentioned service file.
.... aaand looks like we found a bug. The udisks2-zram code is creating config files in /usr/lib64/zram.conf.d on x86_64 yet the service file hardcodes /usr/local/lib/zram.conf.d. Clearly nobody is using that code in reality.

Besides that it creates /usr/lib/modules-load.d/zram.conf and /usr/lib/modprobe.d/zram.conf files (with no indication of being created by UDisks2).

> I'm not sure if this is worth the trouble. The actual process of creating a
> zram
> device is so simple. Instead, if both implementations have configurations,
> one of them should disable itself with a nice message.

Right. There should at least be some degree of awareness since the udisks2-zram module exposes an extra D-Bus interface for management over any existing /dev/zram* block device even when created by a different framework.
Filed upstream: https://github.com/storaged-project/udisks/issues/791

This bug appears to have been reported against 'rawhide' during the Fedora 33 development cycle.
Changing version to 33.

This continues to break installs on arm and aarch64 SBC's. Proposing as a blocker for Fedora 33 Beta, criteria: "The installer must run when launched normally from the release-blocking images."

Created attachment 1711215 [details]
Full boot logs on aarch64 (Fedora-Rawhide-20200811.n.0)

I've pushed udisks2-2.9.1-1.fc33 (bodhi FEDORA-2020-e143a283fb) that brings couple of changes outlined above but probably nothing major that would help to fix this issue. The only notable change is a renamed udisks2-zram-setup@.service unit to help distinguish service origin. I haven't added any Conflicts: for the moment. The rest of the udisks2-zram issues are tracked upstream in the above mentioned ticket.

I've check the comps and the Cockpit spec file, no traces of udisks2-zram Requires:. As far as I know no other package should explicitly require udisks2-zram. Kicking that package from installer image and default install should fix this bugzilla, however the reason it's pulled in needs to be found first. Please reassign as appropriate, this doesn't seem to be a udisks2 bug.

I don't see udisks2-zram package being brought in on either aarch64 or armv7hl image creation. These are from 20200816

https://kojipkgs.fedoraproject.org//work/tasks/9245/49389245/oz-aarch64.log
https://kojipkgs.fedoraproject.org//work/tasks/9171/49389171/mock_output.log

I also only find expected items in the root and initramfs on both
Fedora-Workstation-33-20200816.n.0.aarch64.raw.xz
Fedora-Workstation-armhfp-33-20200816.n.0-sda.raw.xz

So does that mean the original issue is fixed? I don't see anything fishy in the logs either.

I'm not sure, I don't have hardware. In the description it says the problem happens on hardware, not in qemu. That is also strange.

Testing the compose from Fedora-33-20200816.n.0, I am no longer hitting this on arm or aarch64, installs are completed as expected.

Based on comment 21, I'm unproposing this as a Beta blocker, the issue has disappeared. Paul, if it happens again, please nominate this bug again, thanks!

> I'm not sure, I don't have hardware. In the description it says the problem happens on hardware, not in qemu. That is also strange.

I think this must be a false conclusion: there is nothing in either the udev rules or the
unit installed by udisks2-zram that is hardware specific. Most likely there is some race
condition and we just don't happen to see it on qemu.

I installed udisks2-zram on my rpi3, and what happens now is that both zram-generator and udisk2-zram
try to configure the same zram0 device:
Aug 21 01:30:49 rpi3 sudo[2202]:  zbyszek : TTY=pts/0 ; PWD=/home/zbyszek ; USER=root ; COMMAND=/usr/sbin/modprobe -v zram
Aug 21 01:30:49 rpi3 kernel: zram: Added device: zram0
Aug 21 01:30:49 rpi3 systemd[1]: Starting Create swap on /dev/zram0...                    <-------------- this is zram-generator
Aug 21 01:30:49 rpi3 systemd[1]: Starting UDisks2 managed ZRAM device zram0 setup...      <-------------- this is udisk2-zram
Aug 21 01:30:49 rpi3 kernel: zram0: detected capacity change from 0 to 489684992
Aug 21 01:30:49 rpi3 systemd[1]: udisks2-zram-setup: Succeeded.
Aug 21 01:30:49 rpi3 systemd[1]: Finished UDisks2 managed ZRAM device zram0 setup.
Aug 21 01:30:49 rpi3 zram-generator[2219]: Setting up swapspace version 1, size = 467 MiB (489680896 bytes)
Aug 21 01:30:49 rpi3 zram-generator[2219]: no label, UUID=1545938f-92d5-477e-a02d-0e8d5856a56e
Aug 21 01:30:49 rpi3 systemd[1]: Finished Create swap on /dev/zram0.
Aug 21 01:30:50 rpi3 systemd[1]: Activating swap Compressed swap on /dev/zram0...
Aug 21 01:30:50 rpi3 kernel: Adding 478204k swap on /dev/zram0.  Priority:100 extents:1 across:478204k SSFS
Aug 21 01:30:50 rpi3 systemd[1]: Activated swap Compressed swap on /dev/zram0.

The order of messages should not be trusted. They two things are executing in parallel and
happen "at the same time", so we cannot say which one succeeded. udisks2-zram-setup@.service
ignores all errors, while swap-create@.service doesn't, so I think the latter must be winning
the race here, but it's possible that they are both partially active. The kernel allows settings
to be set multiple times before the device is activated.

udisks2-zram-setup@.service will always execute because of the udev rule in
/usr/lib/udev/rules.d/90-udisks2-zram.rules. I don't think we want to have both active,
this can only lead to problems. Should I add a rule to zram-generator to disable itself
if /usr/lib/udev/rules.d/90-udisks2-zram.rules exists? This is ugly, but probably better
than status quo.

>Should I add a rule to zram-generator to disable itself if /usr/lib/udev/rules.d/90-udisks2-zram.rules exists? This is ugly, but probably better than status quo.

I have no strong opinion. The conflict seems unlikely. I don't see udisks2-zram on either armv7hl or aarch64 media, and if it does pop up somewhere on media, that's a bug that definitely needs fixed. But to avoid a problem if someone intentionally installs udisks2-zram? Possibly.

This message is a reminder that Fedora 33 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 33 on 2021-11-30.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '33'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 33 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Fedora 33 changed to end-of-life (EOL) status on 2021-11-30. Fedora 33 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.