Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 2244305 - Fedora Server 39 does not boot on Raspberry Pi 4 (RPi4) from microSD card slot
Summary: Fedora Server 39 does not boot on Raspberry Pi 4 (RPi4) from microSD card slot
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: uboot-tools
Version: 39
Hardware: aarch64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Peter Robinson
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedBlocker
Depends On:
Blocks: F39FinalBlocker
TreeView+ depends on / blocked
 
Reported: 2023-10-15 16:54 UTC by Hristo Marinov
Modified: 2023-11-01 16:12 UTC (History)
16 users (show)

Fixed In Version: uboot-tools-2023.07-3.fc39
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-10-31 21:04:50 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
Shows U-Boot Screen (deleted)
2023-10-16 18:51 UTC, Hristo Marinov
no flags Details
Shows smbiosview output (deleted)
2023-10-17 10:11 UTC, Hristo Marinov
no flags Details

Description Hristo Marinov 2023-10-15 16:54:56 UTC
When I try to boot Fedora Server 39 on Raspberry Pi 4 from the microSD card slot I get the following error on the U-Boot screen:

scanning bus xhci_pci for devices... Unexpected XHCI event TRB, skipping... (3db57620 00000004 01000000 01008401)

Reproducible: Always

Steps to Reproduce:
1. Install Fedora Server 39 on microSD card with arm-image-installer
2. Boot from the Raspberry Pi 4's microSD card slot
Actual Results:  
The following error appears on the U-Boot screen:

scanning bus xhci_pci for devices... Unexpected XHCI event TRB, skipping... (3db57620 00000004 01000000 01008401)

Expected Results:  
The system should boot

I installed the system with the following command:

sudo arm-image-installer --target=rpi4 --image=Fedora-Server-39-20231010.n.0.aarch64.raw.xz --media=/dev/sdc --resizefs --showboot --norootpass --args "nomodeset" --addkey /home/hricky/.ssh/id_ed25519.pub

When I insert the microSD card back into the card reader and plug it into the Raspberry Pi’s USB port, the system boots as expected.

I tested the following images to successfully boot from the SD card slot:
Fedora-Minimal-39-20231010.n.0.aarch64.raw.xz
Fedora-Workstation-39-20231010.n.0.aarch64.raw.xz
Fedora-Server-38-1.6.aarch64.raw.xz

Bootloadrer EEPROM was released on 2023-01-18 and is configured as USB Boot i.e. boot from USB if available, otherwise boot from SD card.

Raspberry Pi 4 Model B, Rev 1.4, 8GB

Comment 1 Kamil Páral 2023-10-16 10:34:14 UTC
Another person confirmed this problem here:
https://discussion.fedoraproject.org/t/fedora-39-server-does-not-boot-on-raspberry-pi-4-from-microsd-card-slot/92755

Proposing for a blocker discussion.

Comment 2 Adam Williamson 2023-10-16 15:20:25 UTC
Could this just be https://bugzilla.redhat.com/show_bug.cgi?id=2241252 , but with an extra error message that happens to be present on your device? Does it work if you boot with 'nomodeset' ?

Comment 3 Peter Robinson 2023-10-16 15:38:54 UTC
> 2. Boot from the Raspberry Pi 4's microSD card slot
> Actual Results:  
> The following error appears on the U-Boot screen:
> 
> scanning bus xhci_pci for devices... Unexpected XHCI event TRB, skipping...
> (3db57620 00000004 01000000 01008401)

That error is a USB error, nothing to do with the mSD, so I'd really need more details.

> sudo arm-image-installer --target=rpi4
> --image=Fedora-Server-39-20231010.n.0.aarch64.raw.xz --media=/dev/sdc
> --resizefs --showboot --norootpass --args "nomodeset" --addkey

Adam: I think he's already answered the nomodeset question.

Comment 4 Adam Williamson 2023-10-16 15:48:06 UTC
oh, whoops, I didn't see that part. so this is a different bug? that would be unfortunate...

Comment 5 Peter Robinson 2023-10-16 15:53:33 UTC
(In reply to Adam Williamson from comment #4)
> oh, whoops, I didn't see that part. so this is a different bug? that would
> be unfortunate...

I think I've already got a fix for that one, it's been discussed upstream, but I'm not sure around the USB crash.

Comment 6 Janne Grunau 2023-10-16 16:19:57 UTC
The asahi downstream u-boot repo (https://github.com/AsahiLinux/u-boot/commits/asahi-releng) has several usb related fixes which might fix this issue. The at least some of the issues were triggered by specific connected usb devices.

Does the issue reproduce without any usb devices connected?

Comment 7 Peter Robinson 2023-10-16 16:50:31 UTC
> scanning bus xhci_pci for devices... Unexpected XHCI event TRB, skipping...
> (3db57620 00000004 01000000 01008401)

I'm seeing this problem on occasion, it's causing me issues bisecting the screen problem, I *think* it's fixed in the upstream 2023.10 GA release.

Comment 8 Peter Robinson 2023-10-16 16:51:19 UTC
> Does the issue reproduce without any usb devices connected?

Yes, I can, what USB controller do you have downstream on asahi?

Comment 9 Janne Grunau 2023-10-16 16:56:01 UTC
Not sure if all issues were controller specific. SoC based usb4/thunderbolt ports use dwc3 for usb3, also present on desktop devices are PCIe xhci controllers from Fresco Logic or Asmedia.

Comment 10 Hristo Marinov 2023-10-16 18:51:05 UTC
Created attachment 1994206 [details]
Shows U-Boot Screen

Not sure if this would be helpful, but here is the short video of what happens after I apply power to the board.

Comment 11 Brandon Nielsen 2023-10-16 20:53:28 UTC
I haven't been able to reproduce with Fedora-Server-39-20231014.n.0.aarch64.raw.xz on my Pi 4B 8 GB. Both uSD and USB boot seem to work consistently.

Comment 12 Adam Williamson 2023-10-17 00:25:06 UTC
Discussed at 2023-10-16 blocker review meeting: https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2023-10-16/f39-blocker-review.2023-10-16-16.00.html . Accepted as a blocker as a clear violation of the requirements for release-blocking images to boot and deploy on Pi 4, which is a blocking platform for aarch64. We decided to take this one on its face for now, as we can adjust the decision later if it turns out this is rare or something.

Comment 13 Hristo Marinov 2023-10-17 10:11:33 UTC
Created attachment 1994294 [details]
Shows smbiosview output

Here's more info about the board provided by the smbiosview command from the UEFI Firmware Shell.

Comment 14 Peter Robinson 2023-10-21 09:18:32 UTC
So upstream has a fix for this, from reports (OpenSUSE) it's not perfect there's also disagreement upstream on the fix (partially from me) but I think the fix will be good enough for most of the Fedora usecases for the short term.

https://koji.fedoraproject.org/koji/taskinfo?taskID=107868656

This build (it's rawhide but will be fine on f39) should have the MMC, and maybe fix the USB. It would be good to get feedback to see if it improves the situation (note this absolutely does NOT fix the screen bug).

Comment 15 Peter Robinson 2023-10-21 09:27:59 UTC
> So upstream has a fix for this, from reports (OpenSUSE) it's not perfect

And others.

Comment 16 Geoffrey Marr 2023-10-23 16:16:48 UTC
Just wanted to add my two-cents here: over the last week, I have booted an RPi4 with SD cards using several different days (20231014, 20231016, and 20231023) F39 images (minimal, server, workstation) and have not been able to trigger this bug. Since last Monday, I've written and booted around 20 microSD cards and haven't had any issues. This is just my experience, but wanted to make this known, should our blocker process get hung up on this bug and we need justification to waive.

Comment 17 Hristo Marinov 2023-10-24 12:33:33 UTC
Unfortunately at least from my side until Fedora-Server-39-20231024.n.0.aarch64.raw.xz the issue is still there.

However, if it comes to a situation where this is the only blocker, I personally think that this could be revisited and possibly overturned.

Comment 18 Janne Grunau 2023-10-25 21:08:48 UTC
I was only able to reproduce a crash in u-boot during xhci init once in ~25 tries. Using a display and not serial console so I can't say with certainty that's the same crash. Using pieeprom-2023-05-11 with USB -> SD boot.

Comment 19 Fedora Update System 2023-10-26 00:40:52 UTC
FEDORA-2023-7994524575 has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-7994524575

Comment 20 Hristo Marinov 2023-10-26 07:54:22 UTC
Fedora-Server-39-1.2.aarch64.raw.xz image boots to login prompt and I can also ssh to the system.

I tested two different brands/sizes of mSD cards several times.

I still need to add a nomodeset parameter to arm-image-installer to show boot messages on the screen, but I prefer it anyway.

Comment 21 Hector Martin 2023-10-26 09:36:15 UTC
That xHCI error sounds exactly like what I fixed here: https://github.com/AsahiLinux/u-boot/commit/8079fe5d87267868381dd1928c70e3d95042f7da

Have the people hitting the bug mentioned what USB devices they have connected? This happens (sometimes, randomly, it's a timing thing) when u-boot tries to stop an endpoint, which usually means something went wrong already, which can be triggered by various other u-boot bugs with other devices.

Honestly though, you'll probably want to try pulling in everything USB-related from our branch and see if that fixes all this pain, because USB is and always has been very broken in u-boot and I fixed like 15 or so bugs here: https://github.com/AsahiLinux/u-boot/commits/asahi-releng

Comment 22 Peter Robinson 2023-10-26 09:59:20 UTC
> Have the people hitting the bug mentioned what USB devices they have
> connected? This happens (sometimes, randomly, it's a timing thing) when
> u-boot tries to stop an endpoint, which usually means something went wrong
> already, which can be triggered by various other u-boot bugs with other
> devices.

No reports, I've seen it with a couple of various upstream points when doing a bisect both with (a Logitech USB dongle) and without devices but it seems to be gone with 23.10 GA.

> Honestly though, you'll probably want to try pulling in everything
> USB-related from our branch and see if that fixes all this pain, because USB
> is and always has been very broken in u-boot and I fixed like 15 or so bugs
> here: https://github.com/AsahiLinux/u-boot/commits/asahi-releng

For the moment we've rolled back to 2023.07, it was more straight forward at this point rather than the game of whack-a-mole that .10 has proven to be.

Do you have a plan around upstreaming them, I don't particularly have an urge to carry a lot of non upstreamed patches for an indeterminate amount of time.

Comment 23 Hector Martin 2023-10-26 12:00:14 UTC
I tried booting about 50 times but couldn't reproduce the bug with uboot-images-armv8-2023.10-0.4.rc3.fc39.noarch. I also tried about 10 times with uboot-images-armv8-2023.10-0.8.fc40.noarch and still no repro.

If I'm right, the bug is a rare race and likely depends on a bunch of random things including possibly clock drift, which is why we're having so much trouble reproing. The best I can offer is this stripped down patch series on top of 2023.10-rc3. If I'm right about the root cause here, this should fix it: https://github.com/AsahiLinux/u-boot/tree/fedora/usb-fixes

Comment 24 Hector Martin 2023-10-26 12:01:42 UTC
I do intend to upstream them but it's on my backburner since right now my hands are full with Fedora Asahi stuff :)

FWIW 2023.07 is equally broken, it probably just doesn't hit the race or only hits it for other people since it is, after all, a race and anything can change the timing.

Comment 25 Peter Robinson 2023-10-26 12:27:11 UTC
> FWIW 2023.07 is equally broken, it probably just doesn't hit the race or
> only hits it for other people since it is, after all, a race and anything
> can change the timing.

Yes, I'm aware of a number of issues with 2023.07 even outside of USB, ATM a bunch of stuff upstream is going on and causing problems, as it stands at the moment I think 2023.07 is the most straight forward solution to unblock the Fedora release. I'm actively working with upstream on a bunch of stuff to hopefully solve some of the core issues we've seen this cycle.

Comment 26 Adam Williamson 2023-10-26 15:38:28 UTC
> I still need to add a nomodeset parameter to arm-image-installer to show boot messages on the screen, but I prefer it anyway.

Hristo, this rather concerns me. Are you saying https://bugzilla.redhat.com/show_bug.cgi?id=2241252 isn't fixed? The main reason we did the revert was to fix that...this one seems less important.

Comment 27 Hristo Marinov 2023-10-26 16:04:09 UTC
(In reply to Adam Williamson from comment #26)
> > I still need to add a nomodeset parameter to arm-image-installer to show boot messages on the screen, but I prefer it anyway.
> 
> Hristo, this rather concerns me. Are you saying
> https://bugzilla.redhat.com/show_bug.cgi?id=2241252 isn't fixed? The main
> reason we did the revert was to fix that...this one seems less important.

Yes, unfortunately https://bugzilla.redhat.com/show_bug.cgi?id=2241252 is there, I just tested it again.

Comment 28 Peter Robinson 2023-10-26 16:18:24 UTC
(In reply to Adam Williamson from comment #26)
> > I still need to add a nomodeset parameter to arm-image-installer to show boot messages on the screen, but I prefer it anyway.
> 
> Hristo, this rather concerns me. Are you saying
> https://bugzilla.redhat.com/show_bug.cgi?id=2241252 isn't fixed? The main
> reason we did the revert was to fix that...this one seems less important.

It is working across my RPi4 devices. Can you post the single line output you get from "dmesg|grep DMI"

Comment 29 Brandon Nielsen 2023-10-26 16:26:10 UTC
(In reply to Peter Robinson from comment #28)
> (In reply to Adam Williamson from comment #26)
> > > I still need to add a nomodeset parameter to arm-image-installer to show boot messages on the screen, but I prefer it anyway.
> > 
> > Hristo, this rather concerns me. Are you saying
> > https://bugzilla.redhat.com/show_bug.cgi?id=2241252 isn't fixed? The main
> > reason we did the revert was to fix that...this one seems less important.
> 
> It is working across my RPi4 devices. Can you post the single line output
> you get from "dmesg|grep DMI"

Barging in here, since I have a 4B that apparently still `needs nomodeset`.

[    0.038253] DMI: raspberrypi,4-model-b Raspberry Pi 4 Model B Rev 1.1/Raspberry Pi 4 Model B Rev 1.1, BIOS 2023.07 07/01/2023

Comment 30 Adam Williamson 2023-10-26 16:31:12 UTC
Sorry, let's keep discussion of that bug in https://bugzilla.redhat.com/show_bug.cgi?id=2241252 . That's the correct bug. This one is something else. I just mentioned it because Hristo mentioned in passing that he needed nomodeset and it made me worry. We should do the follow-up in the other bug.

Comment 31 Adam Williamson 2023-10-26 18:48:27 UTC
Given https://bugzilla.redhat.com/show_bug.cgi?id=2244305#c20 , I'm gonna mark *this* bug as VERIFIED. Other issues can be followed up in other reports.

Comment 32 Hector Martin 2023-10-26 23:33:44 UTC
FYI I just sent the xHCI part of the USB fixes upstream.

Comment 33 Fedora Update System 2023-10-27 02:14:29 UTC
FEDORA-2023-7994524575 has been pushed to the Fedora 39 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-7994524575`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-7994524575

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 34 Fedora Update System 2023-10-30 01:46:55 UTC
FEDORA-2023-7994524575 has been pushed to the Fedora 39 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-7994524575`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-7994524575

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 35 Peter Boy 2023-10-30 19:57:18 UTC
As follow up of our release-blocker-meeting discussion I post my test environment here as well.
Otherwise see https://bugzilla.redhat.com/show_bug.cgi?id=2246871

(1)
My test environment

# cat /etc/os-release 
NAME="Fedora Linux"
VERSION="39 (Server Edition)"
ID=fedora
VERSION_ID=39

# xz  -t Fedora-Server-39-1.2.aarch64.raw.xz 

(no probs detected)

# rpm -q arm-image-installer 
arm-image-installer-3.9-2.fc39.noarch



(2) Write image to disk

arm-image-installer --image=Fedora-Server-39-1.2.aarch64.raw.xz --args "nomodeset"  --media=/dev/mmcblk0  --debug 

[root@rbox assets]# arm-image-installer --image=Fedora-Server-39-1.2.aarch64.raw.xz --args "nomodeset"  --media=/dev/mmcblk0  --debug  
+ shift
+ '[' 0 -gt 0 ']'
++ whoami
+ '[' root '!=' root ']'
+ '[' /dev/mmcblk0 = '' ']'
++ mount
++ grep 'on / '
++ awk '{printf $1"\n"}'
+ ROOTDISK=/dev/mapper/sysvg-root
+ case "$ROOTDISK" in
+ '[' /dev/mmcblk0 = /dev/mapper/sysvg-root ']'
+ '[' '' = cubietruck ']'
+ '[' '' '!=' '' ']'
+ '[' '!' -f Fedora-Server-39-1.2.aarch64.raw.xz ']'
+ '[' '!' -e /dev/mmcblk0 ']'
+ echo ''

+ echo =====================================================
=====================================================
+ '[' -b /dev/fedora/root ']'
+ '[' Fedora-Server-39-1.2.aarch64.raw.xz '!=' '' ']'
+ echo '= Selected Image:                                 '
= Selected Image:                                 
+ echo '= Fedora-Server-39-1.2.aarch64.raw.xz'
= Fedora-Server-39-1.2.aarch64.raw.xz
+ echo '= Selected Media : /dev/mmcblk0'
= Selected Media : /dev/mmcblk0
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' ']'
+ echo =====================================================
=====================================================
+ echo ' '
 
+ echo '*****************************************************'
*****************************************************
+ echo '*****************************************************'
*****************************************************
+ echo '******** WARNING! ALL DATA WILL BE DESTROYED ********'
******** WARNING! ALL DATA WILL BE DESTROYED ********
+ echo '*****************************************************'
*****************************************************
+ echo '*****************************************************'
*****************************************************
+ '[' '' '!=' 1 ']'
+ echo ' '
 
+ echo ' Type '\''YES'\'' to proceed, anything else to exit now '
 Type 'YES' to proceed, anything else to exit now 
+ echo ' '
 
+ printf '= Proceed? '
= Proceed? + read PROCEED
YES
++ echo YES
++ tr '[:lower:]' '[:upper:]'
+ '[' YES '!=' YES ']'
+ umount /dev/mmcblk0 /dev/mmcblk0p1 /dev/mmcblk0p2 /dev/mmcblk0p3
++ fdisk -l /dev/mmcblk0
++ grep LVM
+ '[' '/dev/mmcblk0p3      3328000 124735487 121407488 57.9G 8e Linux LVM' '!=' '' ']'
++ pvs
++ grep /dev/mmcblk0
++ awk '{print $2}'
+ vgchange -a n ''
+ '[' Fedora-Server-39-1.2.aarch64.raw.xz '!=' '' ']'
+ echo '= Writing: '
= Writing: 
+ echo '= Fedora-Server-39-1.2.aarch64.raw.xz '
= Fedora-Server-39-1.2.aarch64.raw.xz 
+ echo '= To: /dev/mmcblk0 ....'
= To: /dev/mmcblk0 ....
+ xzcat Fedora-Server-39-1.2.aarch64.raw.xz
+ dd of=/dev/mmcblk0 oflag=direct bs=4M status=progress iflag=fullblock7507804160 bytes (7.5 GB, 7.0 GiB) copied, 364 s, 20.6 MB/s
1792+0 records in
1792+0 records out
7516192768 bytes (7.5 GB, 7.0 GiB) copied, 364.517 s, 20.6 MB/s
+ sync
+ sleep 3
+ echo '= Writing image complete!'
= Writing image complete!
+ partprobe /dev/mmcblk0
+ '[' -e /dev/mmcblk0p5 ']'
+ '[' -e /dev/mmcblk0p4 ']'
+ '[' -e /dev/mmcblk0p3 ']'
+ export FIRMPART=/dev/mmcblk0p1
+ FIRMPART=/dev/mmcblk0p1
+ BOOTPART=/dev/mmcblk0p2
+ ROOTPART=/dev/mmcblk0p3
+ PARTNUM=3
++ blkid /dev/mmcblk0p3 -s TYPE -o value
+ FS_TYPE=LVM2_member
+ '[' LVM2_member = btrfs ']'
+ '[' '' '!=' '' ']'
+ sleep 5
+ get_lvm_name
++ fdisk -l /dev/mmcblk0
++ grep LVM
+ '[' '/dev/mmcblk0p3      3328000 14680063 11352064  5.4G 8e Linux LVM' '!=' '' ']'
++ pvs
++ grep /dev/mmcblk0
++ awk '{print $2}'
+ LVM_NAME=
+ '[' '' = 1 ']'
+ mkdir /tmp/boot /tmp/root /tmp/fw
+ mount /dev/mmcblk0p2 /tmp/boot
+ '[' 0 -ne 0 ']'
+ mount /dev/mmcblk0p1 /tmp/fw
+ '[' 0 -ne 0 ']'
+ '[' '' = '' ']'
+ get_lvm_name
++ grep LVM
++ fdisk -l /dev/mmcblk0
+ '[' '/dev/mmcblk0p3      3328000 14680063 11352064  5.4G 8e Linux LVM' '!=' '' ']'
++ pvs
++ grep /dev/mmcblk0
++ awk '{print $2}'
+ LVM_NAME=
+ vgchange -a y
++ grep /tmp/root /proc/mounts
+ '[' '' = '' ']'
+ '[' '' '!=' '' ']'
+ mount /dev/mmcblk0p3 /tmp/root
mount: /tmp/root: unknown filesystem type 'LVM2_member'.
       dmesg(1) may have more information after failed mount system call.
++ echo Fedora-Server-39-1.2.aarch64.raw.xz
++ grep IoT
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' ']'
+ '[' '' = 1 ']'
+ '[' '' = 1 ']'
+ PREFIX=/tmp/root
+ '[' '' '!=' '' ']'
+ echo '= No U-boot will be written.'
= No U-boot will be written.
+ TARGET=board
+ '[' '' '!=' '' ']'
+ '[' '' = 1 ']'
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' ']'
+ '[' '' = 1 ']'
+ '[' '' '!=' '' ']'
++ getenforce
+ '[' Enforcing = Disabled ']'
+ '[' '' '!=' '' ']'
+ '[' nomodeset '!=' '' ']'
+ echo '= Adding optional kernel parameters for board : '
= Adding optional kernel parameters for board : 
+ echo '= Parameter: nomodeset'
= Parameter: nomodeset
+ add_kernel_parameter nomodeset
+ '[' -f /tmp/boot/extlinux/extlinux.conf ']'
+ '[' -f /tmp/fw/EFI/fedora/grub.cfg ']'
+ sed -i 's|GRUB_CMDLINE_LINUX="|& nomodeset |' /tmp/root/etc/default/grub
sed: can't read /tmp/root/etc/default/grub: No such file or directory
+ '[' -f /tmp/fw/EFI/fedora/grubenv ']'
+ add_bls_parameter nomodeset
+ for bls in /tmp/boot/loader/entries/*.conf
+ sed -i 's|options|& nomodeset|' /tmp/boot/loader/entries/1df910c164244dc883cf45f9880ac3b9-6.5.6-300.fc39.aarch64.conf
+ '[' board = rpi4 ']'
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' ']'
+ sync
+ umount /tmp/root /dev/mmcblk0p2 /dev/mmcblk0p1
+ '[' '' '!=' '' ']'
+ rmdir /tmp/root /tmp/boot /tmp/fw
+ '[' '' '!=' '' ']'
+ echo ''

+ echo '= Installation Complete! Insert into the board and boot.'
= Installation Complete! Insert into the board and boot.
+ exit 0


(3)
*  Inseert into Rpi2 / 4gb and boot
*  Execute Firstboot
*  Set locale

(4) Basic system works
% ssh pb.158.125
The authenticity of host '192.168.158.125 (192.168.158.125)' can't be established.
ED25519 key fingerprint is SHA256:U4/iqK91pXk/3Q0GcfUVbguYxcWUklE7Bh9YunbDwNo.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.158.125' (ED25519) to the list of known hosts.
pb.158.125's password: 
Web console: https://localhost:9090/ or https://192.168.158.125:9090/

Last login: Mon Oct 30 20:30:11 2023


$ cat /etc/os-release 
NAME="Fedora Linux"
VERSION="39 (Server Edition)"
ID=fedora
VERSION_ID=39
VERSION_CODENAME=""
PLATFORM_ID="platform:f39"
PRETTY_NAME="Fedora Linux 39 (Server Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:39"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f39/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=39
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=39
SUPPORT_END=2024-05-14
VARIANT="Server Edition"
VARIANT_ID=server


$ uname -a 
Linux localhost.localdomain 6.5.6-300.fc39.aarch64 #1 SMP PREEMPT_DYNAMIC Fri Oct  6 19:36:57 UTC 2023 aarch64 GNU/Linux


# lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
mmcblk0         179:0    0 59.5G  0 disk 
├─mmcblk0p1     179:1    0  600M  0 part /boot/efi
├─mmcblk0p2     179:2    0    1G  0 part /boot
└─mmcblk0p3     179:3    0  5.4G  0 part 
  └─fedora-root 253:0    0  5.4G  0 lvm  /
zram0           252:0    0  3.7G  0 disk [SWAP]

expand partition using cfdisk works

lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
mmcblk0         179:0    0 59.5G  0 disk 
├─mmcblk0p1     179:1    0  600M  0 part /boot/efi
├─mmcblk0p2     179:2    0    1G  0 part /boot
└─mmcblk0p3     179:3    0 57.9G  0 part 
  └─fedora-root 253:0    0  5.4G  0 lvm  /
zram0           252:0    0  3.7G  0 disk [SWAP]



(4) LVM administration fails

# pvresize  /dev/mmcblk0p3 
    Cannot use /dev/mmcblk0p3: device is not in devices file
    0 physical volume(s) resized or updated / 0 physical volume(s) not resized

# vgs
    Devices file PVID elqwwdUQpHfHhxMiZvaYfiJUhyU83Vr2 last seen on /dev/vda3 not found.


In Cockpit -> storage the device section is empty.


The output of arm-image-installer is the same for my RockPro64, Roc-rk3399-pc, rock-pi-4a and rock-pi 4b+ But these models don't boot at all, of course (without u-boot)

Comment 36 Fedora Update System 2023-10-31 21:04:50 UTC
FEDORA-2023-7994524575 has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 37 Peter Boy 2023-10-31 21:53:06 UTC
I'm puzzled. That updated in announced in the title as 

arm-image-installer-3.9-2.fc39, bcm283x-firmware-20231017-1.ce3a0b4.fc39, & 1 more

But ii installs 3.8-2 and says inside 3.8.

And the webside says it has been pushed to stable 30 mins ago. And it is the same rpm version as distributed with rc 1.4


Please, can someone shed some light in this?

Comment 38 Adam Williamson 2023-10-31 21:57:06 UTC
I'm puzzled by your comment. What do you mean "But ii installs 3.8-2 and says inside 3.8."?

The build in the update is https://koji.fedoraproject.org/koji/buildinfo?buildID=2313137 . That is 3.9-2. RC-1.4 has 3.9-2 in it: https://kojipkgs.fedoraproject.org/compose/39/Fedora-39-20231031.0/compose/Everything/aarch64/os/Packages/a/arm-image-installer-3.9-2.fc39.noarch.rpm . Where are you saying you're finding 3.8?

Comment 39 Peter Boy 2023-10-31 22:12:58 UTC
I followed the Fedora Update system message in comment 36 from today 1 hour ago (this is highly up to date, I think) and got to this page:

https://bodhi.fedoraproject.org/updates/FEDORA-2023-7994524575

And if I follow the instruction

dnf upgrade --refresh --advisory=FEDORA-2023-7994524575

it gives me (on rc 1.4)

dnf upgrade --refresh --advisory=FEDORA-2023-7994524575 
Fedora 39 - x86_64                                                                        14 kB/s | 7.2 kB     00:00    
Fedora 39 openh264 (From Cisco) - x86_64                                                  11 kB/s | 989  B     00:00    
Fedora 39 - x86_64 - Updates                                                             168 kB/s |  20 kB     00:00    
Dependencies resolved.
Nothing to do.
Complete!


[root@rbox ~]# rpm -qi arm-image-installer 
Name        : arm-image-installer
Version     : 3.8
Release     : 2.fc39
...

and

[root@rbox tmp]# cat /usr/bin/arm-image-installer  | more 
#!/usr/bin/sh

# Copyright (C) 2014-2015 Red Hat Inc.
# SPDX-License-Identifier:  GPL-2.0+

# Automate Media Creation for Fedora ARM
# Current version
VERSION=3.8

# usage message
usage() {
    echo "
Usage: $(basename ${0}) <options>

	--image=IMAGE   - xz compressed image file name
	--media=DEVICE  - media device file (/dev/[sdX|mmcblkX])

Comment 40 Adam Williamson 2023-10-31 22:29:47 UTC
you don't have updates-testing enabled. the update was only pushed stable an hour ago. it won't be in the stable repositories until tomorrow at least.

Comment 41 Peter Boy 2023-10-31 22:33:05 UTC
> RC-1.4 has 3.9-2 in it: https://kojipkgs.fedoraproject.org/compose/39/Fedora-39-20231031.0/compose/Everything/aarch64/os/Packages/a/arm-image-installer-3.9-2.fc39.noarch.rpm . Where are you saying you're finding 3.8?


Now I'm even more puzzled. I downloaded today in the morning Fedora-Server-dvd-x86_64-39-1.4.iso, and tested this including raid installation, VM installation etc.

And then I installed arm-image-installer and got 3.8-2

Comment 42 Adam Williamson 2023-10-31 22:34:40 UTC
That's perfectly normal. Things you install after initial system install come from the mirror repo tree, not from wherever you installed from, so the fact you installed from RC-1.4 doesn't matter.

Comment 43 Peter Boy 2023-10-31 22:36:11 UTC
> you don't have updates-testing enabled. the update was only pushed stable an hour ago. it won't be in the stable repositories until tomorrow at least.

Ah, OK. I see, I have still a lot to learn about our packaging system. So I now install the version from your link and use that for testing.

Comment 44 Adam Williamson 2023-10-31 22:41:10 UTC
for a not-yet-released release, when an update is "pushed stable" all that really means is that it gets tagged 'f39' or 'f40' or whatever, which means it will then be included in the next nightly compose.

the main repo ("fedora" for branched, "rawhide" for Rawhide) on a system running that release points to a mirror system location which is populated by those nightly composes. whenever a compose succeeds, it is "synced out" to mirrors, meaning the contents of that location are updated with the contents of the successful compose.

so, basically, when a pending release update is "pushed stable", systems will only actually see it in the main repo a few hours after the next successful nightly compose happens.

Comment 45 Peter Boy 2023-10-31 23:24:45 UTC
Thanks for the info. 

I just finished testing RPi 4. 

We still have the broken LVM issue, so Server Edition isn't usable on ARM SBCs at all with rc 1.4. It's not an arm-image-installer issue but and disk image issue. So I better continue with the bug I opened, I think.

Comment 46 Peter Robinson 2023-11-01 09:11:26 UTC
> We still have the broken LVM issue, so Server Edition isn't usable on ARM
> SBCs at all with rc 1.4. It's not an arm-image-installer issue but and disk
> image issue. So I better continue with the bug I opened, I think.

I've not seen an issue with LVM on the server images I've been testing on a Rock960. Can we have a dedicated bug for this to try and get to the bottom of the issue you're seeing.

Comment 47 Peter Boy 2023-11-01 10:46:46 UTC
(In reply to Peter Robinson from comment #46)
> > We still have the broken LVM issue, so Server Edition isn't usable on ARM
> > SBCs at all with rc 1.4. It's not an arm-image-installer issue but and disk
> > image issue. So I better continue with the bug I opened, I think.
> 
> I've not seen an issue with LVM on the server images I've been testing on a
> Rock960. 

How did you get that device working? With my RockPro64 I get:

> # arm-image-installer --image=Fedora-Server-39-1.5.aarch64.raw.xz --target=rockpro64-rk3399  --media=/dev/mmcblk0
>
> =====================================================
> = Selected Image:                                 
> = Fedora-Server-39-1.5.aarch64.raw.xz
> = Selected Media : /dev/mmcblk0
> = U-Boot Target : rockpro64-rk3399
> =====================================================
> .....
> = Writing image complete!
> mount: /tmp/root: unknown filesystem type 'LVM2_member'.
>        dmesg(1) may have more information after failed mount system call.
> = No U-Boot files found for rockpro64-rk3399.
> 
> = Installation Complete! Insert into the rockpro64-rk3399 and boot.


And without U-Boot files, it doesn't boot at all.


Well, I saw the issue, Paul Whalen found the cause (https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=2246871#c9) and a possible solution https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=2246871#c11), I tested the fix (deleting the content of etc/lvm/device) and it worked. 

As it looks to me at the moment, we just need someone with commit privilege to the kickstart repo to grep the arm image kickstart file and add a 'rm -f /etc/lvm/devices' and 'rm -f /etc/lvm/cache' to the clean up section at the end and it should work.

 And in the aftermath, we can try to figure out why this happened in F39 and not in F38 and before.



> Can we have a dedicated bug for this to try and get to the bottom
> of the issue you're seeing.

At the moment we have https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=2246871 The title is fine, but the component is wrong.

I can open a new bug, but I don't know which component to select. I found nothing suitable on the list.

Comment 48 Adam Williamson 2023-11-01 16:12:10 UTC
Please don't open any more bugs. https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=2246871 is fine. We can discuss this issue there.


Note You need to log in before you can comment on or make changes to this bug.