Bug 1943983

Description of problem:

I'm attempting to iPXE boot a development F34 kernel (5.11.10-300.fc34.aarch64)
over the internet on a Raspberry Pi 4 B.  The built-in ethernet interface
doesn't come up due to the lack of mdio_bcm_unimac.ko (CONFIG_MDIO_BCM_UNIMAC)
in the current initrd.img:

[    0.000000] Linux version 5.11.10-300.fc34.aarch64 (mockbuild.fedoraproject.org) (gcc (GCC) 11.0.0 20210225 (Red Hat 11.0.0-0), GNU ld version 2.35.1-38.fc34) #1 SMP Thu Mar 25 13:45:38 UTC 2021
[    0.000000] earlycon: uart8250 at MMIO32 0x00000000fe215040 (options '')
[    0.000000] printk: bootconsole [uart8250] enabled
[    0.000000] efi: EFI v2.70 by EDK2
...
[   18.326481] Unable to find mii
[   18.329589] bcmgenet BCM6E4E:00 enabcm6e4ei0: failed to connect to PHY


Is it possible to include this kernel module in the F34 pxeboot initrd
before the release of Fedora 34?


Version-Release number of selected component (if applicable):

Kernel 5.11.10-300.fc34.aarch64 as currently available on
http://ftp.acc.umu.se/mirror/fedora/linux/development/34/Server/aarch64/os/images/pxeboot/


How reproducible:

Always.


Steps to Reproduce:

Prerequisites:

- Raspberry Pi 4 B
- Ethernet connection
- USB mass storage device
- a local HTTP web server (to serve up an uncompressed kernel for iPXE)
- recommended: 3.3V UART serial interface for debugging (e.g. another Raspberry Pi with `enable_uart=1` in its `/boot/config.txt`, `python3-serial` installed, RX/TX/GND wired up  and `miniterm -f direct /dev/serial0 115200`)

1. Ensure that the Raspberry Pi 4 B bootloader firmware is using the most
recent stable release.  This can be achieved by installing and fully upgrading
Raspberry Pi OS.

After a reboot, `rpi-eeprom-update` should report that the boot loader is
up-to-date with the following firmware release:
Thu Mar 18 08:54:11 UTC 2021 (1616057651).

2. Configure the Raspberry Pi 4 to boot from USB MSD before the micro SD card.
Using `raspi-config`, select 6. Advanced options > A6. Boot Order > B2. USB Boot

3. Build a custom release of iPXE for the Raspberry Pi 4 in a suitable
build environment (F32 known to work, F33 known not to).
Within a session of `docker run -it --rm -v /tmp/ipxe:/dist fedora:32`, run:

sudo dnf install -y git-core iasl libuuid-devel make binutils gcc gcc-aarch64-linux-gnu mtools perl python subversion xz-devel
git clone --jobs $(nproc) --recurse-submodules https://github.com/ipxe/pipxe.git
cd pipxe
(cd edk2 && git checkout master && git submodule update --init --recursive)
(cd edk2-non-osi && git checkout master)
(cd edk2-platforms && git checkout master && git submodule update --init --recursive)
sed -i -e s,RPi3,RPi4,g -e s,bin-arm64-efi/rpi.efi,bin-arm64-efi/snp.efi, -e '/git submodule update --init --recursive/d'  Makefile
make -j$(nproc)
cp sdcard.zip /dist

4. Obtain the /tmp/ipxe/sdcard.zip file from the build host and extract it
to a freshly formatted FAT32 partition on the USB MSD device.
The files must be placed on the first EFI System Partition (if it exists)
or the first FAT partition.

5. Download the initrd.img and vmlinuz files and make them available via
a local HTTP server.  iPXE cannot handle compressed Linux kernels, so we need
to decompress the kernel.  On a suitable Linux web server, run something like:

    cd /var/www/html
    curl -LO http://ftp.acc.umu.se/mirror/fedora/linux/development/34/Server/aarch64/os/images/pxeboot/initrd.img
    curl -LO http://ftp.acc.umu.se/mirror/fedora/linux/development/34/Server/aarch64/os/images/pxeboot/vmlinuz
    gzip -dc vmlinuz > vmlinux

Note the web server's IP address as it will be required in the iPXE shell in step 8.

6. Insert the USB MSD to the Raspberry Pi, connect it to the local network
and optionally wire it up for serial console access.
Power it up with or without an SD card.

7. Once the Raspberry Pi logo shows up, use the Esc key to enter the
UEFI boot manager.  In there, select to boot from the USB device
(which contains iPXE as efi/boot/bootaa64.efi).

You might want to delete irrelevant boot options such as UEFI PXE and
UEFI HTTP to ensure that it only attempts to boot from the USB MSD.

8. After iPXE has obtained an IP address, and failed normal PXE boot,
you can use Ctrl-B to drop into the iPXE shell (when the Ctrl-B notice appears).
Tell iPXE to load the initrd image and to boot the uncompressed kernel:

    dhcp
    set u http://192.168.1.42
    initrd ${u}/initrd.img
    chain ${u}/vmlinuz initrd=initrd.img console=tty0 console=ttyS0,115200 panic=10 earlycon=uart8250,mmio32,0xfe215040 inst.text inst.sshd inst.repo=http://ftp.acc.umu.se/mirror/fedora/linux/development/34/Server/aarch64/os/

Refer to the following files for kernel options:

- https://www.raspberrypi.org/documentation/configuration/uart.md
- https://docs.fedoraproject.org/en-US/fedora/f33/install-guide/advanced/Boot_Options/


Actual results:

mdio_bcm_unimac.ko is missing in the output of the following commands:

    mkdir -p /tmp/delete-me && cd /tmp/delete-me
    curl -sL http://ftp.acc.umu.se/mirror/fedora/linux/releases/test/34_Beta/Server/aarch64/os/images/pxeboot/initrd.img |xz -d |cpio -idmv 2>&1 |grep mdio
    curl -sL http://ftp.acc.umu.se/mirror/fedora/linux/development/34/Server/aarch64/os/images/pxeboot/initrd.img |xz -d |cpio -idmv 2>&1 |grep mdio

This leads to a failure to bring up the built-in ethernet interface eth0
on the Raspberry Pi 4 B.  The kernel logs the following messages:

    Unable to find mii
    bcmgenet BCM6E4E:00 enabcm6e4ei0: failed to connect to PHY


Expected results:

The pxeboot initrd.img for Fedora 34 should include the mdio_bcm_unimac.ko
module (CONFIG_MDIO_BCM_UNIMAC) to allow kickstart installations of Raspberry Pi 4 B.


Additional info:

Rawhide ships with the missing mdio_bcm_unimac.ko module:

    mkdir -p /tmp/delete-me && cd /tmp/delete-me
    curl -sL http://ftp.acc.umu.se/mirror/fedora/linux/development/rawhide/Server/aarch64/os/images/pxeboot/initrd.img |xz -d |cpio -idmv 2>&1 |grep mdio

This kernel successfully boots up with iPXE on a Raspberry Pi 4.


Links:

- https://github.com/pftf/RPi4 (Raspberry Pi 4 UEFI Firmware Images [EXPERIMENTAL])
- https://github.com/pftf/RPi4/issues/137 (Linux kernels cannot load network: (failed to connect to PHY) when UEFI netbooted through iPXE or grub)
- https://github.com/ipxe/pipxe (iPXE for the Raspberry Pi)
- https://github.com/ipxe/pipxe/issues/2 (Raspberry Pi 4?)


As an aside, and unrelated to the missing support for CONFIG_MDIO_BCM_UNIMAC
in the current Fedora 34 pxeboot images, I'll also note that newer Linux
kernels are required for SD and WiFi on the Raspberry Pi 4 when booted via UEFI.

As pointed out on https://github.com/pftf/RPi4:

> You will NOT get SD or wireless support in Linux, unless you use a recent Linux kernel (version 5.12 or later) or one into which the 5.12 fixes have been backported as well as a recent Linux wireless firmware package.

Based on changelogs, I believe the relevant fixes by Jeremy Linton should be in
stable kernels released on March 17th, 2021:

- https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.11.7
- https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.10.24

Given this, I'll note that:

- Kernel 5.11.3 is shipped in http://ftp.acc.umu.se/mirror/fedora/linux/releases/test/34_Beta/Server/aarch64/os/images/pxeboot/initrd.img (supports neither SD nor WiFi when booted via UEFI)
- Kernel 5.11.10 is shipped in http://ftp.acc.umu.se/mirror/fedora/linux/releases/test/34_Beta/Server/aarch64/os/images/pxeboot/initrd.img (supports both SD and WiFi when booted via UEFI)