Created attachment 807191 [details]
Log of regenerating initramfs with '--debug'

Proposed Blocker:
"Release-blocking images must boot: All release-blocking images must boot in their supported configurations."

I suspect it's a combination of:
commit 36b2e5e2c27f9e72e9aa40e580b6d9e60799ceae
Author: Colin Walters <walters>
Date:   Wed Sep 11 09:04:45 2013 -0400

    dracut.sh: Fixup previous commit to only read /sys and /proc in hostonly mode
    
    The gnome-ostree build system generates dracut initramfs images on the
    build server, therefore not in hostonly mode.  The build system at the
    moment doesn't mount /sys, and the previous commit caused a hard
    failure due to lack of /sys/devices.
    
    Because we only want /sys/devices in hostonly mode, just move those
    bits inside the hostonly conditional above.

commit 3c4315fa1368f1ee12dfa8abb5dd7c3556da79f8
Author: Harald Hoyer <harald>
Date:   Wed Sep 11 09:57:52 2013 +0200

    dracut-functions.sh: extend module_is_host_only()
    
    If the currently running kernel is not present in the installer root,
    fall back to modalias checking.

since we build the initramfs for the ARM images on one host, in hostonly mode, since each image is board specific (more or less.)

I think I'll hardcode the modules we need for ARM boards entirely in 90kernel-modules.

(In reply to Paul Whalen from comment #1)
> Created attachment 807191 [details]
> Log of regenerating initramfs with '--debug'

Well according to the debug log, "mmc_block" is actually installed.

dracut-install: Handle '/lib/modules/3.11.2-301.fc20.armv7hl/kernel/drivers/mmc/card/mmc_block.ko'
dracut-install: dracut_install('/lib/modules/3.11.2-301.fc20.armv7hl/kernel/drivers/mmc/card/mmc_block.ko', '/lib/modules/3.11.2-301.fc20.armv7hl/kernel/drivers/mmc/card/mmc_block.ko')
dracut-install: dest dir '/var/tmp/initramfs.MXy0ux/lib/modules/3.11.2-301.fc20.armv7hl/kernel/drivers/mmc/card' does not exist
dracut-install: dracut_install('/lib/modules/3.11.2-301.fc20.armv7hl/kernel/drivers/mmc/card', '/lib/modules/3.11.2-301.fc20.armv7hl/kernel/drivers/mmc/card')
dracut-install: dest dir '/var/tmp/initramfs.MXy0ux/lib/modules/3.11.2-301.fc20.armv7hl/kernel/drivers/mmc' does not exist
dracut-install: dracut_install('/lib/modules/3.11.2-301.fc20.armv7hl/kernel/drivers/mmc', '/lib/modules/3.11.2-301.fc20.armv7hl/kernel/drivers/mmc')
dracut-install: mkdir '/var/tmp/initramfs.MXy0ux/lib/modules/3.11.2-301.fc20.armv7hl/kernel/drivers/mmc'
dracut-install: mkdir '/var/tmp/initramfs.MXy0ux/lib/modules/3.11.2-301.fc20.armv7hl/kernel/drivers/mmc/card'
dracut-install: dracut_install ret = 0
dracut-install: cp '/lib/modules/3.11.2-301.fc20.armv7hl/kernel/drivers/mmc/card/mmc_block.ko' '/var/tmp/initramfs.MXy0ux/lib/modules/3.11.2-301.fc20.armv7hl/kernel/drivers/mmc/card/mmc_block.ko'

Discussed this in 2013-10-09 Blocker Review Meeting [1]. Voted an AcceptedBlocker as it violates the following F20 alpha release criterion for ARM images in virt and on bare-metal: "A system installed with a release-blocking desktop must boot to a log in screen where it is possible to log in to a working desktop using a user account created during installation or a 'first boot' utility." [2]

[1] http://meetbot.fedoraproject.org/fedora-blocker-review/2013-10-09/ 
[2] https://fedoraproject.org/wiki/Fedora_20_Alpha_Release_Criteria#Expected_installed_system_boot_behavior

Downloaded:

http://dl.fedoraproject.org/pub/alt/stage/20-Beta-TC2/Images/armhfp/Fedora-Minimal-armhfp-20-Beta-TC2-sda.raw.xz

# unxz ...
# losetup -P -f Fedora-Minimal-armhfp-20-Beta-TC2-sda.raw 
# mount /dev/loop0p1 /mnt/tt
# for i in /mnt/tt/initramfs-3.11.3-301.fc20.armv7hl*; do \
  lsinitrd $i | fgrep mmc_block;done
-rwxr--r--   1 root     root        36164 Oct  3 06:02 usr/lib/modules/3.11.3-301.fc20.armv7hl/kernel/drivers/mmc/card/mmc_block.ko
-rwxr--r--   1 root     root        36172 Oct  3 05:46 usr/lib/modules/3.11.3-301.fc20.armv7hl+lpae/kernel/drivers/mmc/card/mmc_block.ko


Seems to be there..

# for i in /mnt/tt/initramfs-3.11.3-301.fc20.armv7hl*; do lsinitrd $i | fgrep 'with dracut';done
dracut-033-3.git20130913.fc20 with dracut modules:
dracut-033-3.git20130913.fc20 with dracut modules:

Btw, how do you boot this raw image in qemu?

The image creation was missing dracut-config-generic, so we were getting hostonly dracut images in TC1... that should be fixed for TC2. However, we're now also seeing issues resizing the root partition that we weren't before that we need to debug.

regards, Kyle

Hmm, this seems a bit messy now.

The change in TC2 to use dracut-config-generic: is this a fix, or a workaround? Is further work required on that specific issue, or not?

The 'issues resizing the root partition' that Kyle talks about: are those at all related to the initial problem here? If not, they should be filed and tracked separately. If they may block the Beta release, the newly-filed bug should be proposed as a Beta blocker.

dracut-034-8.git20131008.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/dracut-034-8.git20131008.fc20

dracut-034-8.git20131008.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

pulling dracut-config-generic into TC2 was a workaround, though maybe it should be the fix, and arm images just use generic initrds. As we do not know the target device at creation time we need generic, but it would be nice for kernel updates to switch to the host mode and have a smaller initramfs, especially considering that most arm devices are using sdcards for their root storage. the dracut change in dracut-034-8.git20131008.fc20 is not right or sufficient however.

(In reply to Dennis Gilmore from comment #12)
> As we do not know the target device at creation time we need generic…

So, why don't you call dracut with "-N" at generation time and you will have a generic image?

So, now we have:

instmods sdhci_esdhc_imx mmci sdhci_tegra mvsdio omap omapdrm \
         omap_hsmmc panel-tfp410 sdhci_dove ahci_platform pata_imx sata_mv \
         ehci-tegra mmc_block usb_storage


If you need those modules in _every_ arm initramfs, why don't you compile them in the kernel?

If you only need those modules in the shipped install image initramfs, why don't you
a) use the "--no-hostonly" option to build it
or
b) use the --add-drivers "sdhci_esdhc_imx mmci sdhci_tegra mvsdio omap omapdrm \
         omap_hsmmc panel-tfp410 sdhci_dove ahci_platform pata_imx sata_mv \
         ehci-tegra mmc_block usb_storage" to build it?

In any case, why don't you talk to me first before modifying my package?

we do not actually call dracut directly, its called via the kernel install process in appliance-creator. I guess i could patch appliance-creator to regenerate the initramfs. We dont need all of those modules in every arm initramfs but we do need to make sure they are in the ones that dracut makes. I also guess another option is to remove dracut-config-generic in %post in the kickstarts. up until recently dracut had checks that were run, which resulted in drcut determining that it needed to make a generic image. those checks no longer seem to be run. as i said we do not directly run dracut, and id really rather not re run it manually, as then i would need to also re create the u-boot wrapped versions. there is no good way to pass any flags to dracut.


https://fedoraproject.org/wiki/Changes/Virt_ARM_on_x86 gives you an overview on how to run the image on a x86 box.

(In reply to Dennis Gilmore from comment #16)
> up until recently dracut had checks that were run, which resulted in drcut 
> determining that it needed to make a generic image. those checks no longer
> seem to be run.

Well, these checks only check, if /sys /proc /run /dev are mounted properly and are still present.

Anyway, it's not wise to generate a host-only image in the appliance-creator, if these images are used on different hardware.

Removing dracut-config-generic in %post in the kickstarts would probably be the best solution here.

Discussed at 2013-10-16 blocker review meeting: http://meetbot.fedoraproject.org/fedora-blocker-review/2013-10-16/f20beta-blocker-review-4.2013-10-16-16.02.log.txt . This appears to be working, one way or another, in TC4, but we are not sure if the current state is the intended long-term fix, or still a workaround.

Is there a definite plan for how this should be handled in the long term? If so, is it already implemented? Generally, where do we stand on this?

Created attachment 813502 [details]
Fedora 20 Beta TC4 Boot Log - resize success

Created attachment 813506 [details]
Fedora 20 Beta TC5 Boot Log - resize failure

The TC4 used dracut-034-8.git20131008.fc20 and did not include dracut-config-generic. When including the same version of dracut *and* dracut-config-generic in TC5, when root is on sda, the resize of the root filesystem fails and destroys the root partition. Resizing mmc does not work (BZ#1009172), when it does work I would expect the same behaviour. 

I have attached boot logs for TC4 and TC5 with debug, and am unsure which package I should file the bug on. Is this a problem in dracut-modules-growroot - which gives an error on success and failure(in logs), cloud-utils-growpart as the entire partition is destroyed, or a by-product of using a generic config?

(In reply to Paul Whalen from comment #21)
> The TC4 used dracut-034-8.git20131008.fc20 and did not include
> dracut-config-generic. When including the same version of dracut *and*
> dracut-config-generic in TC5, when root is on sda, the resize of the root
> filesystem fails and destroys the root partition. Resizing mmc does not work
> (BZ#1009172), when it does work I would expect the same behaviour. 

So just to confirm here the corruption seen when resizing the sda has been confirmed due to the HW buginess on the Trimslice usb attached SSD. This has been a problem with the device for ever. When tested on highbank with a standard HDD it works fine. I believe kylem has a fix for the resize bug on MMC.

Resize is also causing issues in qemu(BZ#1016648).

If the actual showstopper bug here is fixed by something that's in stable, then we can close it. Shouldn't the resize issue be filed separately?

I'm just going to go ahead and close this now.