Bug 2241274

Created attachment 1991012 [details]
Fedora-Minimal-39_Beta-1.1.aarch64.raw journalctl

Proposing as a blocker for Fedora 39 final, criteria "Release-blocking ARM disk images must boot to the initial-setup utility."

Hmm. The logs seem fairly mum on *why* it failed. Are there any additional anaconda-y logs lying around in /tmp by any chance?

and for the record, this definitely works on a VM, openQA tests it every day and it hasn't failed for months.

So looking at the logs - what might be happening is that Anaconda modules are started that should not be run, such as:

Subscription

Sep 28 20:22:32 fedora org.fedoraproject.Anaconda.Modules.Subscription[971]: INFO:program:Running... systemctl list-unit-files rhsm.service --no-legend
Sep 28 20:22:32 fedora org.fedoraproject.Anaconda.Modules.Timezone[965]: WARNING:py.warnings:/usr/lib/python3.12/site-packages/requests_ftp/ftp.py:9: DeprecationWarning: 'cgi' is deprecated and slated for removal in Python 3.13
Sep 28 20:22:32 fedora org.fedoraproject.Anaconda.Modules.Timezone[965]:   import cgi
Sep 28 20:22:32 fedora org.fedoraproject.Anaconda.Modules.Security[960]: WARNING:py.warnings:/usr/lib/python3.12/site-packages/requests_ftp/ftp.py:9: DeprecationWarning: 'cgi' is deprecated and slated for removal in Python 3.13
Sep 28 20:22:32 fedora org.fedoraproject.Anaconda.Modules.Security[960]:   import cgi
Sep 28 20:22:32 fedora org.fedoraproject.Anaconda.Modules.Subscription[971]: DEBUG:program:Return code: 1
Sep 28 20:22:32 fedora org.fedoraproject.Anaconda.Modules.Subscription[971]: DEBUG:anaconda.modules.subscription.initialization:subscription: The required rhsm systemd service is not available. The Subscription module won't be started.

Payload

Sep 28 20:22:29 fedora org.fedoraproject.Anaconda.Boss[921]: DEBUG:anaconda.modules.boss.module_manager.start_modules:Starting org.fedoraproject.Anaconda.Modules.Payloads.

Storage

Sep 28 20:22:29 fedora org.fedoraproject.Anaconda.Boss[921]: DEBUG:anaconda.modules.boss.module_manager.start_modules:Starting org.fedoraproject.Anaconda.Modules.Storage.

None of those really make sense for Initial Setup & might very well result in the whole thing going down on real hardware.

Digging into it a bit more, it seems like this might be due to a change we did in our config file format and forgot to change it also in Initial Setup (Ooops!).

Basically the old kickstart_modules no longer works and the new thing needs to be used:

https://github.com/rhinstaller/initial-setup/blob/master/data/10-initial-setup.conf#L7

So I'll do that & attach a scratch build with the change shortly, so its possible to check if it really fixes the issue. :)

Discussed during the 2023-10-02 blocker review meeting: [1]

The decision to classify this bug as a AcceptedBlocker (Final) was made:

"We accept this a blocker as it violates the following criterion: "Release-blocking ARM disk images must boot to the initial-setup utility."

[1] https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2023-10-02/f39-blocker-review.2023-10-02-16.01.log.txt

So I think this might fix the issues (provided its the caused by the unwanted modules that try to run):

https://github.com/rhinstaller/initial-setup/pull/148

I've also created a scratch build wit this change:

https://koji.fedoraproject.org/koji/taskinfo?taskID=106994268

At least in my test environment (x86 VM) it seems to correctly start only the expected modules now.
Testing on actual hardware very much welcome! :)

(In reply to Martin Kolman from comment #7)

> 
> I've also created a scratch build wit this change:
> 
> https://koji.fedoraproject.org/koji/taskinfo?taskID=106994268
> 
> At least in my test environment (x86 VM) it seems to correctly start only
> the expected modules now.
> Testing on actual hardware very much welcome! :)

Confirmed, working on the RPi4. Thanks, Martin!

thanks for confirming, Paul - but setting status back to POST so I don't get confused in blocker herding (for blockers, VERIFIED usually means "there's an official update that fixes this and we confirmed it works").

Perfect, thanks for testing - I will do a F39 (+ Rawhide) build shortly. :)

FEDORA-2023-133bdc4283 has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-133bdc4283

FEDORA-2023-133bdc4283 has been pushed to the Fedora 39 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-133bdc4283`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-133bdc4283

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Paul, could you karma the update if it does work? Thanks!

The official fix from comment 12 does not work on Fedora-Minimal-39-20231016.n.0.aarch64.raw.xz, Fedora-Server-39-20231016.n.0.aarch64.raw.xz, or Fedora-Server-39-20231010.n.0.aarch64.raw.xz. The initial-setup utility does not run and shows a "Failed" status in systemctl. However, the scratch build from comment 7 (https://koji.fedoraproject.org/koji/taskinfo?taskID=106994268) does work.

The 0.3.98 tarball does not contain the fix. The fix was definitely committed upstream before 0.3.98 was tagged - https://github.com/rhinstaller/initial-setup/commit/5e3c55df519bc66ab9018bf23504ef3260560517 - but there appear to be no 'official' upstream tarballs. The spec file does not give a URL for the source, nor does it have comments explaining a reproducible procedure for producing the source tarball. So all we know is the 0.3.98 tarball in dist-git was created...somehow...and it's wrong. It has the version numbers bumped and some translation updates, but it does not have the changes to fix this bug.

Martin, can you please fix this up - both the immediate issue, and the process problems here? There's no reason this source tarball should be "magic" like this. Can we not just use the github-generated tag tarballs?

(In reply to Adam Williamson from comment #15)
> The 0.3.98 tarball does not contain the fix. The fix was definitely
> committed upstream before 0.3.98 was tagged -
> https://github.com/rhinstaller/initial-setup/commit/
> 5e3c55df519bc66ab9018bf23504ef3260560517 - but there appear to be no
> 'official' upstream tarballs. The spec file does not give a URL for the
> source, nor does it have comments explaining a reproducible procedure for
> producing the source tarball. So all we know is the 0.3.98 tarball in
> dist-git was created...somehow...and it's wrong. It has the version numbers
> bumped and some translation updates, but it does not have the changes to fix
> this bug.
> 
> Martin, can you please fix this up - both the immediate issue, and the
> process problems here? There's no reason this source tarball should be
> "magic" like this. Can we not just use the github-generated tag tarballs?

Sure, I'll take a look at what is happening. :)

As for using the Github built tarball - there is the usual problem with translations. Translations (and all the associated commit churn) are not part of the initial-setup repo, but are managed by Weblate in a separate repo. So if the plain Github tarball was used, it would not have any translations.

Anaconda solves this by a special github action that automates the translation fetching and automatically creates a proper Github release with the full tarball attached. Its questionable if it makes sense to adapt this substantial amount of automation for Initial Setup, which is released much less frequently - instead of me taking care not to mess it up. :P

FEDORA-2023-56fd65c2b6 has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-56fd65c2b6

(In reply to Fedora Update System from comment #17)
> FEDORA-2023-56fd65c2b6 has been submitted as an update to Fedora 39.
> https://bodhi.fedoraproject.org/updates/FEDORA-2023-56fd65c2b6

This should have the fixed tarball. :)

Thanks mkolman, tested the new build on Fedora-Minimal-39-20231016.n.0.aarch64.raw.xz and had no issues! Works as it should.

FEDORA-2023-56fd65c2b6 has been pushed to the Fedora 39 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-56fd65c2b6`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-56fd65c2b6

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2023-56fd65c2b6 has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.