Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 2241274

Summary: initial-setup text fails on hardware
Product: [Fedora] Fedora Reporter: Paul Whalen <pwhalen>
Component: initial-setupAssignee: Martin Kolman <mkolman>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 39CC: amoloney, awilliam, gmarr, jkonecny, miabbott, mkolman, pbrobinson, robatino, rvykydal, v.podzimek+fedora
Target Milestone: ---   
Target Release: ---   
Hardware: aarch64   
OS: Linux   
Whiteboard: AcceptedBlocker
Fixed In Version: initial-setup-0.3.98-2.fc39 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-10-19 04:41:15 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2143446    
Attachments:
Description Flags
Fedora-Minimal-39_Beta-1.1.aarch64.raw journalctl none

Description Paul Whalen 2023-09-29 00:49:51 UTC
Booting Fedora-Minimal-39_Beta-1.1.aarch64.raw on the Raspberry Pi 4, and Nvidia Jetson Nano initial-setup fails:

systemctl status initial-setup
× initial-setup.service - Initial Setup configuration program
     Loaded: loaded (/usr/lib/systemd/system/initial-setup.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Thu 2023-09-28 20:39:50 EDT; 4min 38s ago
    Process: 804 ExecStartPre=/bin/kill -SIGRTMIN+21 1 (code=exited, status=0/SUCCESS)
    Process: 807 ExecStartPre=/bin/plymouth quit (code=exited, status=0/SUCCESS)
    Process: 810 ExecStart=/usr/libexec/initial-setup/run-initial-setup (code=exited, status=1/FAILURE)
   Main PID: 810 (code=exited, status=1/FAILURE)
        CPU: 48.474s

Sep 28 20:39:49 fedora org.fedoraproject.Anaconda.Boss[962]: DEBUG:anaconda.modules.boss.module_manager.module_manager:org.fedoraproject.Anaconda.Modules.Tim>
Sep 28 20:39:49 fedora org.fedoraproject.Anaconda.Modules.Payloads[1004]: DEBUG:dasbus.connection:Disconnecting from the bus.
Sep 28 20:39:49 fedora org.fedoraproject.Anaconda.Boss[962]: DEBUG:anaconda.modules.boss.module_manager.module_manager:org.fedoraproject.Anaconda.Modules.Pay>
Sep 28 20:39:49 fedora org.fedoraproject.Anaconda.Modules.Localization[1003]: DEBUG:dasbus.connection:Disconnecting from the bus.
Sep 28 20:39:49 fedora org.fedoraproject.Anaconda.Boss[962]: DEBUG:anaconda.modules.boss.module_manager.module_manager:org.fedoraproject.Anaconda.Modules.Loc>
Sep 28 20:39:49 fedora org.fedoraproject.Anaconda.Boss[962]: DEBUG:dasbus.connection:Disconnecting from the bus.
Sep 28 20:39:50 fedora systemd[1]: initial-setup.service: Main process exited, code=exited, status=1/FAILURE
Sep 28 20:39:50 fedora systemd[1]: initial-setup.service: Failed with result 'exit-code'.
Sep 28 20:39:50 fedora systemd[1]: Failed to start initial-setup.service - Initial Setup configuration program.
Sep 28 20:39:50 fedora systemd[1]: initial-setup.service: Consumed 48.474s CPU time.


Reproducible: Always

Steps to Reproduce:
1. Boot Fedora-Minimal-39_Beta-1.1.aarch64.raw on hardware
Actual Results:  
initial-setup fails, boots to login

Expected Results:  
initial-setup

Comment 1 Paul Whalen 2023-09-29 01:23:22 UTC
Created attachment 1991012 [details]
Fedora-Minimal-39_Beta-1.1.aarch64.raw journalctl

Comment 2 Paul Whalen 2023-09-29 15:56:32 UTC
Proposing as a blocker for Fedora 39 final, criteria "Release-blocking ARM disk images must boot to the initial-setup utility."

Comment 3 Adam Williamson 2023-09-29 16:09:30 UTC
Hmm. The logs seem fairly mum on *why* it failed. Are there any additional anaconda-y logs lying around in /tmp by any chance?

Comment 4 Adam Williamson 2023-09-29 16:10:31 UTC
and for the record, this definitely works on a VM, openQA tests it every day and it hasn't failed for months.

Comment 5 Martin Kolman 2023-10-02 14:11:11 UTC
So looking at the logs - what might be happening is that Anaconda modules are started that should not be run, such as:

Subscription

Sep 28 20:22:32 fedora org.fedoraproject.Anaconda.Modules.Subscription[971]: INFO:program:Running... systemctl list-unit-files rhsm.service --no-legend
Sep 28 20:22:32 fedora org.fedoraproject.Anaconda.Modules.Timezone[965]: WARNING:py.warnings:/usr/lib/python3.12/site-packages/requests_ftp/ftp.py:9: DeprecationWarning: 'cgi' is deprecated and slated for removal in Python 3.13
Sep 28 20:22:32 fedora org.fedoraproject.Anaconda.Modules.Timezone[965]:   import cgi
Sep 28 20:22:32 fedora org.fedoraproject.Anaconda.Modules.Security[960]: WARNING:py.warnings:/usr/lib/python3.12/site-packages/requests_ftp/ftp.py:9: DeprecationWarning: 'cgi' is deprecated and slated for removal in Python 3.13
Sep 28 20:22:32 fedora org.fedoraproject.Anaconda.Modules.Security[960]:   import cgi
Sep 28 20:22:32 fedora org.fedoraproject.Anaconda.Modules.Subscription[971]: DEBUG:program:Return code: 1
Sep 28 20:22:32 fedora org.fedoraproject.Anaconda.Modules.Subscription[971]: DEBUG:anaconda.modules.subscription.initialization:subscription: The required rhsm systemd service is not available. The Subscription module won't be started.

Payload

Sep 28 20:22:29 fedora org.fedoraproject.Anaconda.Boss[921]: DEBUG:anaconda.modules.boss.module_manager.start_modules:Starting org.fedoraproject.Anaconda.Modules.Payloads.

Storage

Sep 28 20:22:29 fedora org.fedoraproject.Anaconda.Boss[921]: DEBUG:anaconda.modules.boss.module_manager.start_modules:Starting org.fedoraproject.Anaconda.Modules.Storage.

None of those really make sense for Initial Setup & might very well result in the whole thing going down on real hardware.

Digging into it a bit more, it seems like this might be due to a change we did in our config file format and forgot to change it also in Initial Setup (Ooops!).

Basically the old kickstart_modules no longer works and the new thing needs to be used:

https://github.com/rhinstaller/initial-setup/blob/master/data/10-initial-setup.conf#L7

So I'll do that & attach a scratch build with the change shortly, so its possible to check if it really fixes the issue. :)

Comment 6 Aoife Moloney 2023-10-02 17:21:43 UTC
Discussed during the 2023-10-02 blocker review meeting: [1]

The decision to classify this bug as a AcceptedBlocker (Final) was made:

"We accept this a blocker as it violates the following criterion: "Release-blocking ARM disk images must boot to the initial-setup utility."

[1] https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2023-10-02/f39-blocker-review.2023-10-02-16.01.log.txt

Comment 7 Martin Kolman 2023-10-03 11:39:02 UTC
So I think this might fix the issues (provided its the caused by the unwanted modules that try to run):

https://github.com/rhinstaller/initial-setup/pull/148

I've also created a scratch build wit this change:

https://koji.fedoraproject.org/koji/taskinfo?taskID=106994268

At least in my test environment (x86 VM) it seems to correctly start only the expected modules now.
Testing on actual hardware very much welcome! :)

Comment 8 Paul Whalen 2023-10-05 16:53:14 UTC
(In reply to Martin Kolman from comment #7)

> 
> I've also created a scratch build wit this change:
> 
> https://koji.fedoraproject.org/koji/taskinfo?taskID=106994268
> 
> At least in my test environment (x86 VM) it seems to correctly start only
> the expected modules now.
> Testing on actual hardware very much welcome! :)

Confirmed, working on the RPi4. Thanks, Martin!

Comment 9 Adam Williamson 2023-10-05 17:16:43 UTC
thanks for confirming, Paul - but setting status back to POST so I don't get confused in blocker herding (for blockers, VERIFIED usually means "there's an official update that fixes this and we confirmed it works").

Comment 10 Martin Kolman 2023-10-09 10:00:59 UTC
Perfect, thanks for testing - I will do a F39 (+ Rawhide) build shortly. :)

Comment 11 Fedora Update System 2023-10-09 10:44:08 UTC
FEDORA-2023-133bdc4283 has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-133bdc4283

Comment 12 Fedora Update System 2023-10-10 01:48:49 UTC
FEDORA-2023-133bdc4283 has been pushed to the Fedora 39 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-133bdc4283`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-133bdc4283

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 13 Adam Williamson 2023-10-10 21:49:15 UTC
Paul, could you karma the update if it does work? Thanks!

Comment 14 Geoffrey Marr 2023-10-16 19:46:02 UTC
The official fix from comment 12 does not work on Fedora-Minimal-39-20231016.n.0.aarch64.raw.xz, Fedora-Server-39-20231016.n.0.aarch64.raw.xz, or Fedora-Server-39-20231010.n.0.aarch64.raw.xz. The initial-setup utility does not run and shows a "Failed" status in systemctl. However, the scratch build from comment 7 (https://koji.fedoraproject.org/koji/taskinfo?taskID=106994268) does work.

Comment 15 Adam Williamson 2023-10-16 21:37:26 UTC
The 0.3.98 tarball does not contain the fix. The fix was definitely committed upstream before 0.3.98 was tagged - https://github.com/rhinstaller/initial-setup/commit/5e3c55df519bc66ab9018bf23504ef3260560517 - but there appear to be no 'official' upstream tarballs. The spec file does not give a URL for the source, nor does it have comments explaining a reproducible procedure for producing the source tarball. So all we know is the 0.3.98 tarball in dist-git was created...somehow...and it's wrong. It has the version numbers bumped and some translation updates, but it does not have the changes to fix this bug.

Martin, can you please fix this up - both the immediate issue, and the process problems here? There's no reason this source tarball should be "magic" like this. Can we not just use the github-generated tag tarballs?

Comment 16 Martin Kolman 2023-10-17 13:43:32 UTC
(In reply to Adam Williamson from comment #15)
> The 0.3.98 tarball does not contain the fix. The fix was definitely
> committed upstream before 0.3.98 was tagged -
> https://github.com/rhinstaller/initial-setup/commit/
> 5e3c55df519bc66ab9018bf23504ef3260560517 - but there appear to be no
> 'official' upstream tarballs. The spec file does not give a URL for the
> source, nor does it have comments explaining a reproducible procedure for
> producing the source tarball. So all we know is the 0.3.98 tarball in
> dist-git was created...somehow...and it's wrong. It has the version numbers
> bumped and some translation updates, but it does not have the changes to fix
> this bug.
> 
> Martin, can you please fix this up - both the immediate issue, and the
> process problems here? There's no reason this source tarball should be
> "magic" like this. Can we not just use the github-generated tag tarballs?

Sure, I'll take a look at what is happening. :)

As for using the Github built tarball - there is the usual problem with translations. Translations (and all the associated commit churn) are not part of the initial-setup repo, but are managed by Weblate in a separate repo. So if the plain Github tarball was used, it would not have any translations.

Anaconda solves this by a special github action that automates the translation fetching and automatically creates a proper Github release with the full tarball attached. Its questionable if it makes sense to adapt this substantial amount of automation for Initial Setup, which is released much less frequently - instead of me taking care not to mess it up. :P

Comment 17 Fedora Update System 2023-10-17 15:51:08 UTC
FEDORA-2023-56fd65c2b6 has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-56fd65c2b6

Comment 18 Martin Kolman 2023-10-17 15:52:04 UTC
(In reply to Fedora Update System from comment #17)
> FEDORA-2023-56fd65c2b6 has been submitted as an update to Fedora 39.
> https://bodhi.fedoraproject.org/updates/FEDORA-2023-56fd65c2b6

This should have the fixed tarball. :)

Comment 19 Geoffrey Marr 2023-10-17 18:03:33 UTC
Thanks mkolman, tested the new build on Fedora-Minimal-39-20231016.n.0.aarch64.raw.xz and had no issues! Works as it should.

Comment 20 Fedora Update System 2023-10-18 02:34:51 UTC
FEDORA-2023-56fd65c2b6 has been pushed to the Fedora 39 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-56fd65c2b6`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-56fd65c2b6

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 21 Fedora Update System 2023-10-19 04:41:15 UTC
FEDORA-2023-56fd65c2b6 has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.