Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1943983

Summary: F34: no ethernet on UEFI booted Raspberry Pi 4 due to lack of initramfs MDIO drivers
Product: [Fedora] Fedora Reporter: Throwaway Name <kernel>
Component: dracutAssignee: Anaconda Maintenance Team <anaconda-maint-list>
Status: CLOSED ERRATA QA Contact:
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 34CC: dracut-maint-list, jonathan, pbrobinson, pwhalen, zbyszek
Target Milestone: ---   
Target Release: ---   
Hardware: aarch64   
OS: Linux   
Whiteboard:
Fixed In Version: dracut-053-1.fc34 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-04-01 00:51:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 245418    

Description Throwaway Name 2021-03-28 23:39:03 UTC
Description of problem:

I'm attempting to iPXE boot a development F34 kernel (5.11.10-300.fc34.aarch64)
over the internet on a Raspberry Pi 4 B.  The built-in ethernet interface
doesn't come up due to the lack of mdio_bcm_unimac.ko (CONFIG_MDIO_BCM_UNIMAC)
in the current initrd.img:

[    0.000000] Linux version 5.11.10-300.fc34.aarch64 (mockbuild.fedoraproject.org) (gcc (GCC) 11.0.0 20210225 (Red Hat 11.0.0-0), GNU ld version 2.35.1-38.fc34) #1 SMP Thu Mar 25 13:45:38 UTC 2021
[    0.000000] earlycon: uart8250 at MMIO32 0x00000000fe215040 (options '')
[    0.000000] printk: bootconsole [uart8250] enabled
[    0.000000] efi: EFI v2.70 by EDK2
...
[   18.326481] Unable to find mii
[   18.329589] bcmgenet BCM6E4E:00 enabcm6e4ei0: failed to connect to PHY


Is it possible to include this kernel module in the F34 pxeboot initrd
before the release of Fedora 34?


Version-Release number of selected component (if applicable):

Kernel 5.11.10-300.fc34.aarch64 as currently available on
http://ftp.acc.umu.se/mirror/fedora/linux/development/34/Server/aarch64/os/images/pxeboot/


How reproducible:

Always.


Steps to Reproduce:

Prerequisites:

- Raspberry Pi 4 B
- Ethernet connection
- USB mass storage device
- a local HTTP web server (to serve up an uncompressed kernel for iPXE)
- recommended: 3.3V UART serial interface for debugging (e.g. another Raspberry Pi with `enable_uart=1` in its `/boot/config.txt`, `python3-serial` installed, RX/TX/GND wired up  and `miniterm -f direct /dev/serial0 115200`)

1. Ensure that the Raspberry Pi 4 B bootloader firmware is using the most
recent stable release.  This can be achieved by installing and fully upgrading
Raspberry Pi OS.

After a reboot, `rpi-eeprom-update` should report that the boot loader is
up-to-date with the following firmware release:
Thu Mar 18 08:54:11 UTC 2021 (1616057651).

2. Configure the Raspberry Pi 4 to boot from USB MSD before the micro SD card.
Using `raspi-config`, select 6. Advanced options > A6. Boot Order > B2. USB Boot

3. Build a custom release of iPXE for the Raspberry Pi 4 in a suitable
build environment (F32 known to work, F33 known not to).
Within a session of `docker run -it --rm -v /tmp/ipxe:/dist fedora:32`, run:

sudo dnf install -y git-core iasl libuuid-devel make binutils gcc gcc-aarch64-linux-gnu mtools perl python subversion xz-devel
git clone --jobs $(nproc) --recurse-submodules https://github.com/ipxe/pipxe.git
cd pipxe
(cd edk2 && git checkout master && git submodule update --init --recursive)
(cd edk2-non-osi && git checkout master)
(cd edk2-platforms && git checkout master && git submodule update --init --recursive)
sed -i -e s,RPi3,RPi4,g -e s,bin-arm64-efi/rpi.efi,bin-arm64-efi/snp.efi, -e '/git submodule update --init --recursive/d'  Makefile
make -j$(nproc)
cp sdcard.zip /dist

4. Obtain the /tmp/ipxe/sdcard.zip file from the build host and extract it
to a freshly formatted FAT32 partition on the USB MSD device.
The files must be placed on the first EFI System Partition (if it exists)
or the first FAT partition.

5. Download the initrd.img and vmlinuz files and make them available via
a local HTTP server.  iPXE cannot handle compressed Linux kernels, so we need
to decompress the kernel.  On a suitable Linux web server, run something like:

    cd /var/www/html
    curl -LO http://ftp.acc.umu.se/mirror/fedora/linux/development/34/Server/aarch64/os/images/pxeboot/initrd.img
    curl -LO http://ftp.acc.umu.se/mirror/fedora/linux/development/34/Server/aarch64/os/images/pxeboot/vmlinuz
    gzip -dc vmlinuz > vmlinux

Note the web server's IP address as it will be required in the iPXE shell in step 8.

6. Insert the USB MSD to the Raspberry Pi, connect it to the local network
and optionally wire it up for serial console access.
Power it up with or without an SD card.

7. Once the Raspberry Pi logo shows up, use the Esc key to enter the
UEFI boot manager.  In there, select to boot from the USB device
(which contains iPXE as efi/boot/bootaa64.efi).

You might want to delete irrelevant boot options such as UEFI PXE and
UEFI HTTP to ensure that it only attempts to boot from the USB MSD.

8. After iPXE has obtained an IP address, and failed normal PXE boot,
you can use Ctrl-B to drop into the iPXE shell (when the Ctrl-B notice appears).
Tell iPXE to load the initrd image and to boot the uncompressed kernel:

    dhcp
    set u http://192.168.1.42
    initrd ${u}/initrd.img
    chain ${u}/vmlinuz initrd=initrd.img console=tty0 console=ttyS0,115200 panic=10 earlycon=uart8250,mmio32,0xfe215040 inst.text inst.sshd inst.repo=http://ftp.acc.umu.se/mirror/fedora/linux/development/34/Server/aarch64/os/

Refer to the following files for kernel options:

- https://www.raspberrypi.org/documentation/configuration/uart.md
- https://docs.fedoraproject.org/en-US/fedora/f33/install-guide/advanced/Boot_Options/


Actual results:

mdio_bcm_unimac.ko is missing in the output of the following commands:

    mkdir -p /tmp/delete-me && cd /tmp/delete-me
    curl -sL http://ftp.acc.umu.se/mirror/fedora/linux/releases/test/34_Beta/Server/aarch64/os/images/pxeboot/initrd.img |xz -d |cpio -idmv 2>&1 |grep mdio
    curl -sL http://ftp.acc.umu.se/mirror/fedora/linux/development/34/Server/aarch64/os/images/pxeboot/initrd.img |xz -d |cpio -idmv 2>&1 |grep mdio

This leads to a failure to bring up the built-in ethernet interface eth0
on the Raspberry Pi 4 B.  The kernel logs the following messages:

    Unable to find mii
    bcmgenet BCM6E4E:00 enabcm6e4ei0: failed to connect to PHY


Expected results:

The pxeboot initrd.img for Fedora 34 should include the mdio_bcm_unimac.ko
module (CONFIG_MDIO_BCM_UNIMAC) to allow kickstart installations of Raspberry Pi 4 B.


Additional info:

Rawhide ships with the missing mdio_bcm_unimac.ko module:

    mkdir -p /tmp/delete-me && cd /tmp/delete-me
    curl -sL http://ftp.acc.umu.se/mirror/fedora/linux/development/rawhide/Server/aarch64/os/images/pxeboot/initrd.img |xz -d |cpio -idmv 2>&1 |grep mdio

This kernel successfully boots up with iPXE on a Raspberry Pi 4.


Links:

- https://github.com/pftf/RPi4 (Raspberry Pi 4 UEFI Firmware Images [EXPERIMENTAL])
- https://github.com/pftf/RPi4/issues/137 (Linux kernels cannot load network: (failed to connect to PHY) when UEFI netbooted through iPXE or grub)
- https://github.com/ipxe/pipxe (iPXE for the Raspberry Pi)
- https://github.com/ipxe/pipxe/issues/2 (Raspberry Pi 4?)


As an aside, and unrelated to the missing support for CONFIG_MDIO_BCM_UNIMAC
in the current Fedora 34 pxeboot images, I'll also note that newer Linux
kernels are required for SD and WiFi on the Raspberry Pi 4 when booted via UEFI.

As pointed out on https://github.com/pftf/RPi4:

> You will NOT get SD or wireless support in Linux, unless you use a recent Linux kernel (version 5.12 or later) or one into which the 5.12 fixes have been backported as well as a recent Linux wireless firmware package.

Based on changelogs, I believe the relevant fixes by Jeremy Linton should be in
stable kernels released on March 17th, 2021:

- https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.11.7
- https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.10.24

Given this, I'll note that:

- Kernel 5.11.3 is shipped in http://ftp.acc.umu.se/mirror/fedora/linux/releases/test/34_Beta/Server/aarch64/os/images/pxeboot/initrd.img (supports neither SD nor WiFi when booted via UEFI)
- Kernel 5.11.10 is shipped in http://ftp.acc.umu.se/mirror/fedora/linux/releases/test/34_Beta/Server/aarch64/os/images/pxeboot/initrd.img (supports both SD and WiFi when booted via UEFI)

Comment 1 Throwaway Name 2021-03-29 09:05:01 UTC
In #fedora-arm, Nicolas Chauvet confirmed that CONFIG_MDIO_BCM_UNIMAC=m is indeed set in the F34 kernel (fedpkg clone kernel -a) and suggested that the lack of mdio_bcm_unimac.ko in the initramfs could be caused by a missing directory in https://github.com/dracutdevs/dracut/tree/master/modules.d/90kernel-network-modules.

https://src.fedoraproject.org/rpms/dracut suggests that Fedora 34 currently ships dracut 051-1 (spec: https://src.fedoraproject.org/rpms/dracut/blob/f34/f/dracut.spec).

The more recent dracut 052 release (https://github.com/dracutdevs/dracut/releases/tag/052) includes https://github.com/dracutdevs/dracut/commit/3c8ca29, which would likely resolve the issue reported:

>  fix(kernel-network-modules): also install modules from mdio subdirectory 
> 
> A recent change in Linux kernel v5.10 split the MDIO drivers from the PHY
> subdirectory into a new MDIO subdirectory [0]. This caused MDIO drivers
> to not be included in the initramfs.
> 
> Also include the modules of this new subdirectory to prevent this issue.
> 
> [0]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=a9770eac511

This might explain why the current Rawhide, which ships dracut 053, initramfs has the necessary MDIO network drivers whereas the F34 initramfs lacks most of them.  I can confirm that the Raspberry Pi 4 successfully boots with functional built-in ethernet on the Rawhide kernel and initramfs.

I'm therefore changing the bug report's component from anaconda-images to dracut and ask:

Is it possible to backport https://github.com/dracutdevs/dracut/commit/3c8ca29 into https://src.fedoraproject.org/rpms/dracut/tree/f34 ?

Comment 2 Peter Robinson 2021-03-29 13:20:15 UTC
This was due to a change in the upstream kernel, it was fixed in this commit and in dracut 052+ https://github.com/dracutdevs/dracut/pull/1020/commits/c8be886786d5a28d711ef37532b9e9f88a91f8dd

I thought we had this in F-34 already, bit it seems it was only build for F-35 :-/

Comment 3 Fedora Update System 2021-03-29 13:29:56 UTC
FEDORA-2021-2af4af2b59 has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-2af4af2b59

Comment 4 Fedora Update System 2021-03-30 14:38:05 UTC
FEDORA-2021-2af4af2b59 has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-2af4af2b59`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-2af4af2b59

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 5 Fedora Update System 2021-04-01 00:51:58 UTC
FEDORA-2021-2af4af2b59 has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.