1016648 – ARM guest on x86_64 fails to boot, mount: special device /dev/vda3 does not exist

Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1016648 - ARM guest on x86_64 fails to boot, mount: special device /dev/vda3 does not exist

Summary: ARM guest on x86_64 fails to boot, mount: special device /dev/vda3 does not e...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Fedora
Classification:	Fedora
Component:	cloud-initramfs-tools
Sub Component:
Version:	20
Hardware:	arm
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	---
Assignee:	Juerg Haefliger
QA Contact:	Fedora Extras Quality Assurance
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	ARMTracker
TreeView+	depends on / blocked

Reported:	2013-10-08 13:20 UTC by Alexander Todorov
Modified:	2013-12-21 02:09 UTC (History)
CC List:	8 users (show)
Fixed In Version:	cloud-initramfs-tools-0.20-2.fc19
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2013-11-26 04:51:36 UTC
Type:	Bug
Embargoed:
Dependent Products:

Attachments	(Terms of Use)
/var/log/libvirt/qemu/fedora20-arm-3.log (1.28 KB, text/plain) 2013-10-08 13:23 UTC, Alexander Todorov	no flags	Details
View All

Description Alexander Todorov 2013-10-08 13:20:04 UTC

Description of problem:

I'm following this test case
http://fedoraproject.org/wiki/QA:Testcase_Virt_x86_on_ARM

using Fedora-Minimal-armhfp-20-Beta-TC2-sda.raw image

serial console shows:
[    9.029080] systemd[1]: Starting Journal Service...
         Starting Journal Service...
[  OK  ] Started Journal Service.
[    9.095873] systemd[1]: Started Journal Service.
         Starting Create list of required static device nodes...rrent kernel...
[  OK  ] Listening on udev Kernel Socket.
[  OK  ] Listening on udev Control Socket.
[  OK  ] Reached target Sockets.
         Starting Setup Virtual Console...
[  OK  ] Reached target Swap.
[    9.839364] systemd-journald[58]: Vacuuming done, freed 0 bytes
[  OK  ] Reached target Local File Systems.
[  OK  ] Started Setup Virtual Console.
[  OK  ] Started Create list of required static device nodes ...current kernel.
         Starting Create static device nodes in /dev...
[  OK  ] Started Create static device nodes in /dev.
[  OK  ] Started dracut cmdline hook.
         Starting dracut pre-udev hook...
[  OK  ] Started dracut pre-udev hook.
         Starting udev Kernel Device Manager...
[  OK  ] Started udev Kernel Device Manager.
[   17.654429] systemd-udevd[186]: starting version 208
         Starting dracut pre-trigger hook...
[  OK  ] Started dracut pre-trigger hook.
         Starting udev Coldplug all Devices...
[  OK  ] Started udev Coldplug all Devices.
         Starting dracut initqueue hook...
         Starting Show Plymouth Boot Screen...
[  OK  ] Reached target System Initialization.
[   27.207769]  vda: vda1 vda2 vda3
[  OK  ] Started dracut initqueue hook.
         Starting dracut pre-mount hook...
[  OK  ] Started Show Plymouth Boot Screen.
[  OK  ] Reached target Paths.
[  OK  ] Reached target Basic System.
[  OK  ] Started dracut pre-mount hook.
         Mounting /sysroot...
[   30.378797] EXT4-fs (vda3): mounted filesystem with ordered data mode. Opts: (null)
[  OK  ] Mounted /sysroot.
[  OK  ] Reached target Initrd Root File System.
         Starting Reload Configuration from the Real Root...
[  OK  ] Started Reload Configuration from the Real Root.
[  OK  ] Reached target Initrd File Systems.
[  OK  ] Reached target Initrd Default Target.
dracut-pre-pivot[312]: # === old sfdisk -d ===
dracut-pre-pivot[312]: # partition table of /dev/vda
dracut-pre-pivot[312]: unit: sectors
dracut-pre-pivot[312]: /dev/vda1 : start=     1953, size=  1000001, Id=83
dracut-pre-pivot[312]: /dev/vda2 : start=  1001954, size=   250000, Id=83
dracut-pre-pivot[312]: /dev/vda3 : start=  1251954, size=  2734375, Id=83
dracut-pre-pivot[312]: /dev/vda4 : start=        0, size=        0, Id= 0
dracut-pre-pivot[312]: # === new sfdisk -d ===
dracut-pre-pivot[312]: # partition table of /dev/vda
dracut-pre-pivot[312]: unit: sectors
dracut-pre-pivot[312]: /dev/vda1 : start=     1953, size=  1000001, Id=83
dracut-pre-pivot[312]: /dev/vda2 : start=  1001954, size=   250000, Id=83
dracut-pre-pivot[312]: /dev/vda3 : start=  1251954, size=  2925198, Id=83
dracut-pre-pivot[312]: /dev/vda4 : start=        0, size=        0, Id= 0
[   37.414070]  vda: vda1 vda2 vda3
dracut-pre-pivot[312]: mount: special device /dev/vda3 does not exist
dracut-pre-pivot[312]: growroot Fatal: Failed to re-mount /dev/vda3, this is bad



Version-Release number of selected component (if applicable):
dracut-033-3.git20130913.fc20.armv7hl.rpm 
Fedora 20, Beta-TC2

How reproducible:
always

Steps to Reproduce:
0. My host is x86_64 with latest Fedora 20 packages (but I guess that doesn't matter much in this case).
1. Decompress the raw image and extract the /boot directory as described in the test case.
2. Place them under /var/lib/libvirt/images and restore their SELinux context if necessary
3. Using virt-manager wizard create a new guest with the following parameters
  - CPU arch: arm
  - CPU type: vexpress a9
  - select the NON PAE vmlinuz and initramfs images
  - select the vexpress-v2p-ca9.dtb DTB image
  - select the raw disk image
  - specify kernel parameters: console=ttyAMA0 rw root=/dev/vda3
  - specify OS type as Linux/Fedora 20

Actual results:
Guest boots and I can see the system initializing. Then it fails to mount the root fs and hangs.

Expected results:
Everything works.

Additional info:

The last section of the console log looks surprising. It shows vda3 present and then mount fails:

dracut-pre-pivot[312]: # === old sfdisk -d ===
dracut-pre-pivot[312]: # partition table of /dev/vda
dracut-pre-pivot[312]: unit: sectors
dracut-pre-pivot[312]: /dev/vda1 : start=     1953, size=  1000001, Id=83
dracut-pre-pivot[312]: /dev/vda2 : start=  1001954, size=   250000, Id=83
dracut-pre-pivot[312]: /dev/vda3 : start=  1251954, size=  2734375, Id=83
dracut-pre-pivot[312]: /dev/vda4 : start=        0, size=        0, Id= 0
dracut-pre-pivot[312]: # === new sfdisk -d ===
dracut-pre-pivot[312]: # partition table of /dev/vda
dracut-pre-pivot[312]: unit: sectors
dracut-pre-pivot[312]: /dev/vda1 : start=     1953, size=  1000001, Id=83
dracut-pre-pivot[312]: /dev/vda2 : start=  1001954, size=   250000, Id=83
dracut-pre-pivot[312]: /dev/vda3 : start=  1251954, size=  2925198, Id=83
dracut-pre-pivot[312]: /dev/vda4 : start=        0, size=        0, Id= 0
[   37.414070]  vda: vda1 vda2 vda3
dracut-pre-pivot[312]: mount: special device /dev/vda3 does not exist
dracut-pre-pivot[312]: growroot Fatal: Failed to re-mount /dev/vda3, this is bad



The guest XML looks like this:

<domain type='qemu'>
  <name>fedora20-arm-3</name>
  <uuid>7591ab7f-f835-4195-944e-1632ca46f3ae</uuid>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='armv7l' machine='vexpress-a9'>hvm</type>
    <kernel>/var/lib/libvirt/images/arm-minimal/boot/vmlinuz-3.11.3-301.fc20.armv7hl</kernel>
    <initrd>/var/lib/libvirt/images/arm-minimal/boot/initramfs-3.11.3-301.fc20.armv7hl.img</initrd>
    <cmdline>console=ttyAMA0 rw root=/dev/vda3</cmdline>
    <dtb>/var/lib/libvirt/images/arm-minimal/boot/dtb-3.11.3-301.fc20.armv7hl/vexpress-v2p-ca9.dtb</dtb>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-arm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/arm-minimal/Fedora-Minimal-armhfp-20-Beta-TC2-sda.raw'/>
      <target dev='vda' bus='virtio'/>
      <address type='virtio-mmio'/>
    </disk>
    <interface type='network'>
      <mac address='52:54:00:e1:a5:5c'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='virtio-mmio'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
  </devices>
</domain>

^^ this XML appears to be correct

Comment 1 Alexander Todorov 2013-10-08 13:23:24 UTC

Created attachment 809279 [details]
/var/log/libvirt/qemu/fedora20-arm-3.log

the guest log

Comment 2 Harald Hoyer 2013-10-08 15:22:20 UTC

I don't know which script calls growroot in the dracut pre pivot hook. This script has to be fixed. It's not dracut.

Please assign to the correct component.

Thank you!

What is the output of:

# lsinitrd /var/lib/libvirt/images/arm-minimal/boot/initramfs-3.11.3-301.fc20.armv7hl.img | fgrep pre-pivot

Comment 3 Harald Hoyer 2013-10-08 15:26:11 UTC

seems to be dracut-modules-growroot, which seems to be part of cloud-initramfs-tools

Comment 4 Harald Hoyer 2013-10-09 06:41:34 UTC

Hmm, I think growroot should _not_ use udevsettle() the shell function, but call udevadm settle directly.

i=0
while :; do
  udevadm settle --timeout=100000 --exit-if-exists="${rootdev}"
  [ -e "${rootdev}" ] && break
  [ $i -ge 100 ] && break
  i=$(($i+1))
done

Comment 5 Juerg Haefliger 2013-10-14 14:36:14 UTC

This used to work with the Fedora-Minimal-armhfp-20-Alpha-4-sda image. Beta-TC2 is behaving differently.

This is weird. vda3 doesn't come back after growpart resized the partition:

dracut-pre-pivot[313]: # === old sfdisk -d ===
dracut-pre-pivot[313]: # partition table of /dev/vda
dracut-pre-pivot[313]: unit: sectors
dracut-pre-pivot[313]: /dev/vda1 : start=     1953, size=  1000001, Id=83
dracut-pre-pivot[313]: /dev/vda2 : start=  1001954, size=   250000, Id=83
dracut-pre-pivot[313]: /dev/vda3 : start=  1251954, size=  3466494, Id=83
dracut-pre-pivot[313]: /dev/vda4 : start=        0, size=        0, Id= 0
dracut-pre-pivot[313]: # === new sfdisk -d ===
dracut-pre-pivot[313]: # partition table of /dev/vda
dracut-pre-pivot[313]: unit: sectors
dracut-pre-pivot[313]: /dev/vda1 : start=     1953, size=  1000001, Id=83
dracut-pre-pivot[313]: /dev/vda2 : start=  1001954, size=   250000, Id=83
dracut-pre-pivot[313]: /dev/vda3 : start=  1251954, size=  3571326, Id=83
dracut-pre-pivot[313]: /dev/vda4 : start=        0, size=        0, Id= 0
dracut-pre-pivot[313]: mount: special device /dev/vda3 does not exist
dracut-pre-pivot[313]: growroot Fatal: Failed to re-mount /dev/vda3, this is bad
dracut-pre-pivot[313]: Warning: /dev/vda3 does not exist
Warning: /dev/vda3 does not exist

<SNIP>

dracut:/# ls -l /dev/vda*
brw-rw---- 1 root disk 252, 0 Oct 14 14:26 /dev/vda
brw-rw---- 1 root disk 252, 1 Oct 14 14:26 /dev/vda1
brw-rw---- 1 root disk 252, 2 Oct 14 14:26 /dev/vda2

The partition table is valid though:

dracut:/# partx --show /dev/vda
NR   START     END SECTORS   SIZE NAME UUID
 1    1953 1001953 1000001 488.3M      0002d63e-01
 2 1001954 1251953  250000 122.1M      0002d63e-02
 3 1251954 4823279 3571326   1.7G      0002d63e-03

And the 3rd partition can be added manually:

dracut:/# partx --add --nr 3 /dev/vda
dracut:/# [  305.919783] EXT4-fs (vda3): mounted filesystem with ordered data mode. Opts: (null)

I admit I don't fully understand how this works. Questions:
1) Why is there a different behaviour between Alpha-4 and Beta-TC2?
2) Why is vda3 automatically remounted when the partition is added? Is that systemd?

Comment 6 Paul Whalen 2013-10-22 13:58:46 UTC

This appeared in TC2 when we added dracut-config-generic to the images. This was a work around for another bug(BZ#1015234), in TC1 we got host only initrd's. 

Resize does not work at all on mmc so not affected (BZ#1009172), but I have also found it on the Trimslice when using sda.

Comment 7 Fedora Update System 2013-11-22 13:10:11 UTC

cloud-initramfs-tools-0.20-2.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/cloud-initramfs-tools-0.20-2.fc20

Comment 8 Fedora Update System 2013-11-22 13:28:13 UTC

cloud-initramfs-tools-0.20-2.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/cloud-initramfs-tools-0.20-2.fc19

Comment 9 Fedora Update System 2013-11-22 13:40:02 UTC

cloud-initramfs-tools-0.20-2.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/cloud-initramfs-tools-0.20-2.el6

Comment 10 Fedora Update System 2013-11-23 02:38:46 UTC

Package cloud-initramfs-tools-0.20-2.el6:
* should fix your issue,
* was pushed to the Fedora EPEL 6 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=epel-testing cloud-initramfs-tools-0.20-2.el6'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-EPEL-2013-12173/cloud-initramfs-tools-0.20-2.el6
then log in and leave karma (feedback).

Comment 11 Fedora Update System 2013-11-26 04:51:36 UTC

cloud-initramfs-tools-0.20-2.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 12 Fedora Update System 2013-12-20 16:52:42 UTC

cloud-initramfs-tools-0.20-2.el6 has been pushed to the Fedora EPEL 6 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 13 Fedora Update System 2013-12-21 02:09:01 UTC

cloud-initramfs-tools-0.20-2.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.