Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1160429
Summary: | Timed out waiting for device dev-mapper-mpatha2.device | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Mark Hamzy <hamzy> | ||||||||||||
Component: | device-mapper-multipath | Assignee: | Ben Marzinski <bmarzins> | ||||||||||||
Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||||||||
Severity: | urgent | Docs Contact: | |||||||||||||
Priority: | urgent | ||||||||||||||
Version: | 21 | CC: | agk, anaconda-maint-list, bmarzins, dan, dlehman, dwysocha, fdinitto, g.kaviyarasu, gustavold, hannsj_uhl, heinzm, jbrassow, jonathan, lvm-team, msnitzer, prajnoha, prockai, robatino, vanmeeuwen+fedora | ||||||||||||
Target Milestone: | --- | ||||||||||||||
Target Release: | --- | ||||||||||||||
Hardware: | ppc64 | ||||||||||||||
OS: | Linux | ||||||||||||||
Whiteboard: | |||||||||||||||
Fixed In Version: | device-mapper-multipath-0.4.9-68.fc21.1 | Doc Type: | Bug Fix | ||||||||||||
Doc Text: | Story Points: | --- | |||||||||||||
Clone Of: | Environment: | ||||||||||||||
Last Closed: | 2014-11-17 06:31:53 UTC | Type: | Bug | ||||||||||||
Regression: | --- | Mount Type: | --- | ||||||||||||
Documentation: | --- | CRM: | |||||||||||||
Verified Versions: | Category: | --- | |||||||||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||
Embargoed: | |||||||||||||||
Bug Depends On: | |||||||||||||||
Bug Blocks: | 1071880, 1043129, 1146697 | ||||||||||||||
Attachments: |
|
Description
Mark Hamzy
2014-11-04 19:27:37 UTC
[root@junostac4 ~]# pvscan PV /dev/mapper/mpathc3 VG fedora-server_junostac4 lvm2 [558.41 GiB / 60.00 MiB free] PV /dev/mapper/mpathb1 VG fedora-server_junostac4 lvm2 [558.91 GiB / 0 free] PV /dev/mapper/mpatha1 VG fedora-server_junostac4 lvm2 [558.91 GiB / 0 free] Total: 3 [1.64 TiB] / in use: 3 [1.64 TiB] / in no VG: 0 [0 ] [root@junostac4 ~]# lvscan ACTIVE '/dev/fedora-server_junostac4/swap' [4.00 GiB] inherit ACTIVE '/dev/fedora-server_junostac4/home' [1.58 TiB] inherit ACTIVE '/dev/fedora-server_junostac4/root' [50.00 GiB] inherit [root@junostac4 ~]# fdisk -l /dev/sda ... Device Boot Start End Sectors Size Id Type /dev/sda1 2048 1172121599 1172119552 558.9G 8e Linux LVM It seems that /boot now lives on sdc2 instead of sda2. [root@junostac4 ~]# fdisk -l /dev/sdc ... Device Boot Start End Sectors Size Id Type /dev/sdc1 * 2048 10239 8192 4M 41 PPC PReP Boot /dev/sdc2 10240 1034239 1024000 500M 83 Linux /dev/sdc3 1034240 1172121599 1171087360 558.4G 8e Linux LVM [root@junostac4 ~]# fdisk -l /dev/sdf ... Device Boot Start End Sectors Size Id Type /dev/sdf1 * 2048 10239 8192 4M 41 PPC PReP Boot /dev/sdf2 10240 1034239 1024000 500M 83 Linux /dev/sdf3 1034240 1172121599 1171087360 558.4G 8e Linux LVM [root@junostac4 ~]# mkdir /mnt/tmp [root@junostac4 ~]# mount /dev/mapper/mpathc2 /mnt/tmp [ 4521.662362] EXT4-fs (dm-5): mounted filesystem with ordered data mode. Opts: (null) [root@junostac4 ~]# ls /mnt/tmp config-3.17.0-301.fc21.ppc64p7 grub2 initramfs-0-rescue-7003d14bbf9d4260a5939f2c2613e633.img initramfs-3.17.0-301.fc21.ppc64p7.img initrd-plymouth.img lost+found System.map-3.17.0-301.fc21.ppc64p7 vmlinuz-0-rescue-7003d14bbf9d4260a5939f2c2613e633 vmlinuz-3.17.0-301.fc21.ppc64p7 [root@junostac4 ~]# sed -i -e 's,mpatha2,mpathc2,' /etc/fstab [root@junostac4 ~]# reboot Created attachment 953790 [details]
anaconda.log
Created attachment 953791 [details]
program.log
Created attachment 953792 [details]
storage.log
[root@junostac4 ~]# cat /etc/multipath/wwids # Multipath wwids, Version : 1.0 # NOTE: This file is automatically maintained by multipath and multipathd. # You should not need to edit this file in normal circumstances. # # Valid WWIDs: /3500003937809e810/ /350000393780939e4/ /35000039378093b9c/ [root@junostac4 ~]# cat /etc/multipath/bindings # Multipath bindings, Version : 1.0 # NOTE: this file is automatically maintained by the multipath program. # You should not need to edit this file in normal circumstances. # # Format: # alias wwid # mpatha 3500003937809e810 mpathb 350000393780939e4 mpathc 35000039378093b9c Created attachment 953816 [details]
syslog
So it seems that, instead of modifying /etc/fstab, I should have modified /etc/multipath/bindings. So I rebooted back into the installer and now it seems that /boot lives on sda. [anaconda root@junostac4 ~]# cat /etc/multipath/wwids # Multipath wwids, Version : 1.0 # NOTE: This file is automatically maintained by multipath and multipathd. # You should not need to edit this file in normal circumstances. # # Valid WWIDs: /3500003937809e810/ /350000393780939e4/ /35000039378093b9c/ [anaconda root@junostac4 ~]# cat /etc/multipath/bindings # Multipath bindings, Version : 1.0 # NOTE: this file is automatically maintained by the multipath program. # You should not need to edit this file in normal circumstances. # # Format: # alias wwid # mpatha 3500003937809e810 mpathb 350000393780939e4 mpathc 35000039378093b9c [anaconda root@junostac4 ~]# fdisk -l /dev/mapper/mpatha Disk /dev/mapper/mpatha: 558.9 GiB, 600127266816 bytes, 1172123568 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x189bbec5 Device Boot Start End Sectors Size Id Type /dev/mapper/mpatha1 * 2048 10239 8192 4M 41 PPC PReP Boot /dev/mapper/mpatha2 10240 1034239 1024000 500M 83 Linux /dev/mapper/mpatha3 1034240 1172121599 1171087360 558.4G 8e Linux LVM [anaconda root@junostac4 ~]# fdisk -l /dev/mapper/mpathc Disk /dev/mapper/mpathc: 558.9 GiB, 600127266816 bytes, 1172123568 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0xe8e2d638 Device Boot Start End Sectors Size Id Type /dev/mapper/mpathc1 2048 1172121599 1172119552 558.9G 8e Linux LVM [anaconda root@junostac4 ~]# multipath -ll mpathc (3500003937809e810) dm-5 IBM ,MBF2600RC size=559G features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 0:0:4:0 sda 8:0 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 1:0:4:0 sdd 8:48 active ready running mpathb (350000393780939e4) dm-1 IBM ,MBF2600RC size=559G features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 0:0:5:0 sdb 8:16 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 1:0:5:0 sde 8:64 active ready running mpatha (35000039378093b9c) dm-0 IBM ,MBF2600RC size=559G features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 0:0:7:0 sdc 8:32 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 1:0:7:0 sdf 8:80 active ready running [anaconda root@junostac4 ~]# mkdir /mnt/tmp [anaconda root@junostac4 ~]# mount /dev/mapper/mpatha2 /mnt/tmp [anaconda root@junostac4 ~]# ls /mnt/tmp System.map-3.17.0-301.fc21.ppc64p7 initrd-plymouth.img config-3.17.0-301.fc21.ppc64p7 lost+found grub2 vmlinuz-0-rescue-7003d14bbf9d4260a5939f2c2613e633 initramfs-0-rescue-7003d14bbf9d4260a5939f2c2613e633.img vmlinuz-3.17.0-301.fc21.ppc64p7 initramfs-3.17.0-301.fc21.ppc64p7.img [anaconda root@junostac4 ~]# multipath -v3 Nov 04 20:22:42 | loading /lib64/multipath/libcheckdirectio.so checker Nov 04 20:22:42 | loading /lib64/multipath/libprioconst.so prioritizer Nov 04 20:22:42 | sda: not found in pathvec Nov 04 20:22:42 | sda: mask = 0x3f Nov 04 20:22:42 | sda: dev_t = 8:0 Nov 04 20:22:42 | sda: size = 1172123568 Nov 04 20:22:42 | sda: vendor = IBM Nov 04 20:22:42 | sda: product = MBF2600RC Nov 04 20:22:42 | sda: rev = 610A Nov 04 20:22:42 | sda: h:b:t:l = 0:0:4:0 Nov 04 20:22:42 | sda: path state = running Nov 04 20:22:42 | sda: 24018 cyl, 128 heads, 32 sectors/track, start at 0 Nov 04 20:22:42 | sda: serial = EA09PB809DSB Nov 04 20:22:42 | sda: get_state Nov 04 20:22:42 | sda: path checker = directio (internal default) Nov 04 20:22:42 | sda: checker timeout = 30000 ms (sysfs setting) Nov 04 20:22:42 | directio: starting new request Nov 04 20:22:42 | directio: io finished 4096/0 Nov 04 20:22:42 | sda: state = up Nov 04 20:22:42 | sda: uid_attribute = ID_SERIAL (internal default) Nov 04 20:22:42 | sda: uid = 3500003937809e810 (udev) Nov 04 20:22:42 | sda: detect_prio = 1 (config file default) Nov 04 20:22:42 | sda: prio = const (internal default) Nov 04 20:22:42 | sda: prio = (internal default) Nov 04 20:22:42 | sda: const prio = 1 Nov 04 20:22:42 | sdb: not found in pathvec Nov 04 20:22:42 | sdb: mask = 0x3f Nov 04 20:22:42 | sdb: dev_t = 8:16 Nov 04 20:22:42 | sdb: size = 1172123568 Nov 04 20:22:42 | sdb: vendor = IBM Nov 04 20:22:42 | sdb: product = MBF2600RC Nov 04 20:22:42 | sdb: rev = 610A Nov 04 20:22:42 | sdb: h:b:t:l = 0:0:5:0 Nov 04 20:22:42 | sdb: path state = running Nov 04 20:22:42 | sdb: 24018 cyl, 128 heads, 32 sectors/track, start at 0 Nov 04 20:22:42 | sdb: serial = EA09PB809C26 Nov 04 20:22:42 | sdb: get_state Nov 04 20:22:42 | sdb: path checker = directio (internal default) Nov 04 20:22:42 | sdb: checker timeout = 30000 ms (sysfs setting) Nov 04 20:22:42 | directio: starting new request Nov 04 20:22:42 | directio: io finished 4096/0 Nov 04 20:22:42 | sdb: state = up Nov 04 20:22:42 | sdb: uid_attribute = ID_SERIAL (internal default) Nov 04 20:22:42 | sdb: uid = 350000393780939e4 (udev) Nov 04 20:22:42 | sdb: detect_prio = 1 (config file default) Nov 04 20:22:42 | sdb: prio = const (internal default) Nov 04 20:22:42 | sdb: prio = (internal default) Nov 04 20:22:42 | sdb: const prio = 1 Nov 04 20:22:42 | sdc: not found in pathvec Nov 04 20:22:42 | sdc: mask = 0x3f Nov 04 20:22:42 | sdc: dev_t = 8:32 Nov 04 20:22:42 | sdc: size = 1172123568 Nov 04 20:22:42 | sdc: vendor = IBM Nov 04 20:22:42 | sdc: product = MBF2600RC Nov 04 20:22:42 | sdc: rev = 610A Nov 04 20:22:42 | sdc: h:b:t:l = 0:0:7:0 Nov 04 20:22:42 | sdc: path state = running Nov 04 20:22:42 | sdc: 24018 cyl, 128 heads, 32 sectors/track, start at 0 Nov 04 20:22:42 | sdc: serial = EA09PB809C52 Nov 04 20:22:42 | sdc: get_state Nov 04 20:22:42 | sdc: path checker = directio (internal default) Nov 04 20:22:42 | sdc: checker timeout = 30000 ms (sysfs setting) Nov 04 20:22:42 | directio: starting new request Nov 04 20:22:42 | directio: io finished 4096/0 Nov 04 20:22:42 | sdc: state = up Nov 04 20:22:42 | sdc: uid_attribute = ID_SERIAL (internal default) Nov 04 20:22:42 | sdc: uid = 35000039378093b9c (udev) Nov 04 20:22:42 | sdc: detect_prio = 1 (config file default) Nov 04 20:22:42 | sdc: prio = const (internal default) Nov 04 20:22:42 | sdc: prio = (internal default) Nov 04 20:22:42 | sdc: const prio = 1 Nov 04 20:22:42 | sdd: not found in pathvec Nov 04 20:22:42 | sdd: mask = 0x3f Nov 04 20:22:42 | sdd: dev_t = 8:48 Nov 04 20:22:42 | sdd: size = 1172123568 Nov 04 20:22:42 | sdd: vendor = IBM Nov 04 20:22:42 | sdd: product = MBF2600RC Nov 04 20:22:42 | sdd: rev = 610A Nov 04 20:22:42 | sdd: h:b:t:l = 1:0:4:0 Nov 04 20:22:42 | sdd: path state = running Nov 04 20:22:42 | sdd: 24018 cyl, 128 heads, 32 sectors/track, start at 0 Nov 04 20:22:42 | sdd: serial = EA09PB809DSB Nov 04 20:22:42 | sdd: get_state Nov 04 20:22:42 | sdd: path checker = directio (internal default) Nov 04 20:22:42 | sdd: checker timeout = 30000 ms (sysfs setting) Nov 04 20:22:42 | directio: starting new request Nov 04 20:22:42 | directio: io finished 4096/0 Nov 04 20:22:42 | sdd: state = up Nov 04 20:22:42 | sdd: uid_attribute = ID_SERIAL (internal default) Nov 04 20:22:42 | sdd: uid = 3500003937809e810 (udev) Nov 04 20:22:42 | sdd: detect_prio = 1 (config file default) Nov 04 20:22:42 | sdd: prio = const (internal default) Nov 04 20:22:42 | sdd: prio = (internal default) Nov 04 20:22:42 | sdd: const prio = 1 Nov 04 20:22:42 | sde: not found in pathvec Nov 04 20:22:42 | sde: mask = 0x3f Nov 04 20:22:42 | sde: dev_t = 8:64 Nov 04 20:22:42 | sde: size = 1172123568 Nov 04 20:22:42 | sde: vendor = IBM Nov 04 20:22:42 | sde: product = MBF2600RC Nov 04 20:22:42 | sde: rev = 610A Nov 04 20:22:42 | sde: h:b:t:l = 1:0:5:0 Nov 04 20:22:42 | sde: path state = running Nov 04 20:22:42 | sde: 24018 cyl, 128 heads, 32 sectors/track, start at 0 Nov 04 20:22:42 | sde: serial = EA09PB809C26 Nov 04 20:22:42 | sde: get_state Nov 04 20:22:42 | sde: path checker = directio (internal default) Nov 04 20:22:42 | sde: checker timeout = 30000 ms (sysfs setting) Nov 04 20:22:42 | directio: starting new request Nov 04 20:22:42 | directio: io finished 4096/0 Nov 04 20:22:42 | sde: state = up Nov 04 20:22:42 | sde: uid_attribute = ID_SERIAL (internal default) Nov 04 20:22:42 | sde: uid = 350000393780939e4 (udev) Nov 04 20:22:42 | sde: detect_prio = 1 (config file default) Nov 04 20:22:42 | sde: prio = const (internal default) Nov 04 20:22:42 | sde: prio = (internal default) Nov 04 20:22:42 | sde: const prio = 1 Nov 04 20:22:42 | sdf: not found in pathvec Nov 04 20:22:42 | sdf: mask = 0x3f Nov 04 20:22:42 | sdf: dev_t = 8:80 Nov 04 20:22:42 | sdf: size = 1172123568 Nov 04 20:22:42 | sdf: vendor = IBM Nov 04 20:22:42 | sdf: product = MBF2600RC Nov 04 20:22:42 | sdf: rev = 610A Nov 04 20:22:42 | sdf: h:b:t:l = 1:0:7:0 Nov 04 20:22:42 | sdf: path state = running Nov 04 20:22:42 | sdf: 24018 cyl, 128 heads, 32 sectors/track, start at 0 Nov 04 20:22:42 | sdf: serial = EA09PB809C52 Nov 04 20:22:42 | sdf: get_state Nov 04 20:22:42 | sdf: path checker = directio (internal default) Nov 04 20:22:42 | sdf: checker timeout = 30000 ms (sysfs setting) Nov 04 20:22:42 | directio: starting new request Nov 04 20:22:42 | directio: io finished 4096/0 Nov 04 20:22:42 | sdf: state = up Nov 04 20:22:42 | sdf: uid_attribute = ID_SERIAL (internal default) Nov 04 20:22:42 | sdf: uid = 35000039378093b9c (udev) Nov 04 20:22:42 | sdf: detect_prio = 1 (config file default) Nov 04 20:22:42 | sdf: prio = const (internal default) Nov 04 20:22:42 | sdf: prio = (internal default) Nov 04 20:22:42 | sdf: const prio = 1 Nov 04 20:22:42 | sr0: device node name blacklisted Nov 04 20:22:42 | loop0: device node name blacklisted Nov 04 20:22:42 | loop1: device node name blacklisted Nov 04 20:22:42 | loop2: device node name blacklisted Nov 04 20:22:42 | dm-0: device node name blacklisted Nov 04 20:22:42 | dm-1: device node name blacklisted Nov 04 20:22:42 | dm-2: device node name blacklisted Nov 04 20:22:42 | dm-3: device node name blacklisted Nov 04 20:22:42 | dm-4: device node name blacklisted Nov 04 20:22:42 | dm-5: device node name blacklisted Nov 04 20:22:42 | dm-6: device node name blacklisted Nov 04 20:22:42 | dm-7: device node name blacklisted Nov 04 20:22:42 | dm-8: device node name blacklisted Nov 04 20:22:42 | dm-9: device node name blacklisted ===== paths list ===== uuid hcil dev dev_t pri dm_st chk_st vend/prod/rev 3500003937809e810 0:0:4:0 sda 8:0 1 undef ready IBM ,MBF2600RC 350000393780939e4 0:0:5:0 sdb 8:16 1 undef ready IBM ,MBF2600RC 35000039378093b9c 0:0:7:0 sdc 8:32 1 undef ready IBM ,MBF2600RC 3500003937809e810 1:0:4:0 sdd 8:48 1 undef ready IBM ,MBF2600RC 350000393780939e4 1:0:5:0 sde 8:64 1 undef ready IBM ,MBF2600RC 35000039378093b9c 1:0:7:0 sdf 8:80 1 undef ready IBM ,MBF2600RC Nov 04 20:22:42 | params = 0 0 2 1 service-time 0 1 2 8:0 1 1 service-time 0 1 2 8:48 1 1 Nov 04 20:22:42 | status = 2 0 0 0 2 1 A 0 1 2 8:0 A 0 0 1 E 0 1 2 8:48 A 0 0 1 Nov 04 20:22:42 | mpathc: disassemble map [0 0 2 1 service-time 0 1 2 8:0 1 1 service-time 0 1 2 8:48 1 1 ] Nov 04 20:22:42 | mpathc: disassemble status [2 0 0 0 2 1 A 0 1 2 8:0 A 0 0 1 E 0 1 2 8:48 A 0 0 1 ] Nov 04 20:22:42 | params = 0 0 2 1 service-time 0 1 2 8:16 1 1 service-time 0 1 2 8:64 1 1 Nov 04 20:22:42 | status = 2 0 0 0 2 1 A 0 1 2 8:16 A 0 0 1 E 0 1 2 8:64 A 0 0 1 Nov 04 20:22:42 | mpathb: disassemble map [0 0 2 1 service-time 0 1 2 8:16 1 1 service-time 0 1 2 8:64 1 1 ] Nov 04 20:22:42 | mpathb: disassemble status [2 0 0 0 2 1 A 0 1 2 8:16 A 0 0 1 E 0 1 2 8:64 A 0 0 1 ] Nov 04 20:22:42 | params = 0 0 2 1 service-time 0 1 2 8:32 1 1 service-time 0 1 2 8:80 1 1 Nov 04 20:22:42 | status = 2 0 0 0 2 1 A 0 1 2 8:32 A 0 0 1 E 0 1 2 8:80 A 0 0 1 Nov 04 20:22:42 | mpatha: disassemble map [0 0 2 1 service-time 0 1 2 8:32 1 1 service-time 0 1 2 8:80 1 1 ] Nov 04 20:22:42 | mpatha: disassemble status [2 0 0 0 2 1 A 0 1 2 8:32 A 0 0 1 E 0 1 2 8:80 A 0 0 1 ] Nov 04 20:22:42 | directio checker refcount 6 Nov 04 20:22:42 | const prioritizer refcount 6 Nov 04 20:22:42 | directio checker refcount 5 Nov 04 20:22:42 | const prioritizer refcount 5 Nov 04 20:22:43 | directio checker refcount 4 Nov 04 20:22:43 | const prioritizer refcount 4 Nov 04 20:22:43 | directio checker refcount 3 Nov 04 20:22:43 | const prioritizer refcount 3 Nov 04 20:22:43 | directio checker refcount 2 Nov 04 20:22:43 | const prioritizer refcount 2 Nov 04 20:22:43 | directio checker refcount 1 Nov 04 20:22:43 | const prioritizer refcount 1 Nov 04 20:22:43 | unloading const prioritizer Nov 04 20:22:43 | unloading directio checker What is your /etc/fstab? I just reinstalled. And now /dev/mapper/mpathb gets the drive where /boot exists! [root@junostac4 ~]# cat /etc/fstab # # /etc/fstab # Created by anaconda on Tue Nov 4 17:42:33 2014 # # Accessible filesystems, by reference, are maintained under '/dev/disk' # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info # /dev/mapper/fedora--server_junostac4-root / ext4 defaults 1 1 /dev/mapper/mpatha2 /boot ext4 defaults 1 2 /dev/mapper/fedora--server_junostac4-home /home ext4 defaults 1 2 /dev/mapper/fedora--server_junostac4-swap swap swap defaults 0 0 [root@junostac4 ~]# ls -l /dev/mapper/ total 0 crw-------. 1 root root 10, 236 Nov 4 18:36 control lrwxrwxrwx. 1 root root 8 Nov 4 18:36 fedora--server_junostac4-home -> ../dm-10 lrwxrwxrwx. 1 root root 7 Nov 4 18:36 fedora--server_junostac4-root -> ../dm-9 lrwxrwxrwx. 1 root root 7 Nov 4 18:36 fedora--server_junostac4-swap -> ../dm-8 lrwxrwxrwx. 1 root root 7 Nov 4 18:36 mpatha -> ../dm-0 lrwxrwxrwx. 1 root root 7 Nov 4 18:36 mpatha1 -> ../dm-1 lrwxrwxrwx. 1 root root 7 Nov 4 18:36 mpathb -> ../dm-4 lrwxrwxrwx. 1 root root 7 Nov 4 18:36 mpathb1 -> ../dm-5 lrwxrwxrwx. 1 root root 7 Nov 4 18:36 mpathb2 -> ../dm-6 lrwxrwxrwx. 1 root root 7 Nov 4 18:36 mpathb3 -> ../dm-7 lrwxrwxrwx. 1 root root 7 Nov 4 18:36 mpathc -> ../dm-2 lrwxrwxrwx. 1 root root 7 Nov 4 18:36 mpathc1 -> ../dm-3 [root@junostac4 ~]# cat /etc/multipath/bindings # Multipath bindings, Version : 1.0 # NOTE: this file is automatically maintained by the multipath program. # You should not need to edit this file in normal circumstances. # # Format: # alias wwid # mpatha 3500003937809e810 mpathb 350000393780939e4 mpathc 35000039378093b9c [root@junostac4 ~]# multipath -ll mpathc (35000039378093b9c) dm-2 IBM ,MBF2600RC size=559G features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 0:0:7:0 sdc 8:32 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 1:0:7:0 sdf 8:80 active ready running mpathb (350000393780939e4) dm-4 IBM ,MBF2600RC size=559G features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 0:0:5:0 sdb 8:16 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 1:0:5:0 sde 8:64 active ready running mpatha (3500003937809e810) dm-0 IBM ,MBF2600RC size=559G features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 0:0:4:0 sda 8:0 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 1:0:4:0 sdd 8:48 active ready running [root@junostac4 ~]# multipath -v3 Nov 04 18:59:46 | loading /lib64/multipath/libcheckdirectio.so checker Nov 04 18:59:46 | loading /lib64/multipath/libprioconst.so prioritizer Nov 04 18:59:46 | sda: not found in pathvec Nov 04 18:59:46 | sda: mask = 0x3f Nov 04 18:59:46 | sda: dev_t = 8:0 Nov 04 18:59:46 | sda: size = 1172123568 Nov 04 18:59:46 | sda: vendor = IBM Nov 04 18:59:46 | sda: product = MBF2600RC Nov 04 18:59:46 | sda: rev = 610A Nov 04 18:59:46 | sda: h:b:t:l = 0:0:4:0 Nov 04 18:59:46 | sda: path state = running Nov 04 18:59:46 | sda: 24018 cyl, 128 heads, 32 sectors/track, start at 0 Nov 04 18:59:46 | sda: serial = EA09PB809DSB Nov 04 18:59:46 | sda: get_state Nov 04 18:59:46 | sda: path checker = directio (internal default) Nov 04 18:59:46 | sda: checker timeout = 30000 ms (sysfs setting) Nov 04 18:59:46 | directio: starting new request Nov 04 18:59:46 | directio: io finished 4096/0 Nov 04 18:59:46 | sda: state = up Nov 04 18:59:46 | sda: uid_attribute = ID_SERIAL (internal default) Nov 04 18:59:46 | sda: uid = 3500003937809e810 (udev) Nov 04 18:59:46 | sda: detect_prio = 1 (config file default) Nov 04 18:59:46 | sda: prio = const (internal default) Nov 04 18:59:46 | sda: prio = (internal default) Nov 04 18:59:46 | sda: const prio = 1 Nov 04 18:59:46 | sdb: not found in pathvec Nov 04 18:59:46 | sdb: mask = 0x3f Nov 04 18:59:46 | sdb: dev_t = 8:16 Nov 04 18:59:46 | sdb: size = 1172123568 Nov 04 18:59:46 | sdb: vendor = IBM Nov 04 18:59:46 | sdb: product = MBF2600RC Nov 04 18:59:46 | sdb: rev = 610A Nov 04 18:59:46 | sdb: h:b:t:l = 0:0:5:0 Nov 04 18:59:46 | sdb: path state = running Nov 04 18:59:46 | sdb: 24018 cyl, 128 heads, 32 sectors/track, start at 0 Nov 04 18:59:46 | sdb: serial = EA09PB809C26 Nov 04 18:59:46 | sdb: get_state Nov 04 18:59:46 | sdb: path checker = directio (internal default) Nov 04 18:59:46 | sdb: checker timeout = 30000 ms (sysfs setting) Nov 04 18:59:46 | directio: starting new request Nov 04 18:59:46 | directio: io finished 4096/0 Nov 04 18:59:46 | sdb: state = up Nov 04 18:59:46 | sdb: uid_attribute = ID_SERIAL (internal default) Nov 04 18:59:46 | sdb: uid = 350000393780939e4 (udev) Nov 04 18:59:46 | sdb: detect_prio = 1 (config file default) Nov 04 18:59:46 | sdb: prio = const (internal default) Nov 04 18:59:46 | sdb: prio = (internal default) Nov 04 18:59:46 | sdb: const prio = 1 Nov 04 18:59:46 | sdc: not found in pathvec Nov 04 18:59:46 | sdc: mask = 0x3f Nov 04 18:59:46 | sdc: dev_t = 8:32 Nov 04 18:59:46 | sdc: size = 1172123568 Nov 04 18:59:46 | sdc: vendor = IBM Nov 04 18:59:46 | sdc: product = MBF2600RC Nov 04 18:59:46 | sdc: rev = 610A Nov 04 18:59:46 | sdc: h:b:t:l = 0:0:7:0 Nov 04 18:59:46 | sdc: path state = running Nov 04 18:59:46 | sdc: 24018 cyl, 128 heads, 32 sectors/track, start at 0 Nov 04 18:59:46 | sdc: serial = EA09PB809C52 Nov 04 18:59:46 | sdc: get_state Nov 04 18:59:46 | sdc: path checker = directio (internal default) Nov 04 18:59:46 | sdc: checker timeout = 30000 ms (sysfs setting) Nov 04 18:59:46 | directio: starting new request Nov 04 18:59:46 | directio: io finished 4096/0 Nov 04 18:59:46 | sdc: state = up Nov 04 18:59:46 | sdc: uid_attribute = ID_SERIAL (internal default) Nov 04 18:59:46 | sdc: uid = 35000039378093b9c (udev) Nov 04 18:59:46 | sdc: detect_prio = 1 (config file default) Nov 04 18:59:46 | sdc: prio = const (internal default) Nov 04 18:59:46 | sdc: prio = (internal default) Nov 04 18:59:46 | sdc: const prio = 1 Nov 04 18:59:46 | sdd: not found in pathvec Nov 04 18:59:46 | sdd: mask = 0x3f Nov 04 18:59:46 | sdd: dev_t = 8:48 Nov 04 18:59:46 | sdd: size = 1172123568 Nov 04 18:59:46 | sdd: vendor = IBM Nov 04 18:59:46 | sdd: product = MBF2600RC Nov 04 18:59:46 | sdd: rev = 610A Nov 04 18:59:46 | sdd: h:b:t:l = 1:0:4:0 Nov 04 18:59:46 | sdd: path state = running Nov 04 18:59:46 | sdd: 24018 cyl, 128 heads, 32 sectors/track, start at 0 Nov 04 18:59:46 | sdd: serial = EA09PB809DSB Nov 04 18:59:46 | sdd: get_state Nov 04 18:59:46 | sdd: path checker = directio (internal default) Nov 04 18:59:46 | sdd: checker timeout = 30000 ms (sysfs setting) Nov 04 18:59:46 | directio: starting new request Nov 04 18:59:46 | directio: io finished 4096/0 Nov 04 18:59:46 | sdd: state = up Nov 04 18:59:46 | sdd: uid_attribute = ID_SERIAL (internal default) Nov 04 18:59:46 | sdd: uid = 3500003937809e810 (udev) Nov 04 18:59:46 | sdd: detect_prio = 1 (config file default) Nov 04 18:59:46 | sdd: prio = const (internal default) Nov 04 18:59:46 | sdd: prio = (internal default) Nov 04 18:59:46 | sdd: const prio = 1 Nov 04 18:59:46 | sde: not found in pathvec Nov 04 18:59:46 | sde: mask = 0x3f Nov 04 18:59:46 | sde: dev_t = 8:64 Nov 04 18:59:46 | sde: size = 1172123568 Nov 04 18:59:46 | sde: vendor = IBM Nov 04 18:59:46 | sde: product = MBF2600RC Nov 04 18:59:46 | sde: rev = 610A Nov 04 18:59:46 | sde: h:b:t:l = 1:0:5:0 Nov 04 18:59:46 | sde: path state = running Nov 04 18:59:46 | sde: 24018 cyl, 128 heads, 32 sectors/track, start at 0 Nov 04 18:59:46 | sde: serial = EA09PB809C26 Nov 04 18:59:46 | sde: get_state Nov 04 18:59:46 | sde: path checker = directio (internal default) Nov 04 18:59:46 | sde: checker timeout = 30000 ms (sysfs setting) Nov 04 18:59:46 | directio: starting new request Nov 04 18:59:46 | directio: io finished 4096/0 Nov 04 18:59:46 | sde: state = up Nov 04 18:59:46 | sde: uid_attribute = ID_SERIAL (internal default) Nov 04 18:59:46 | sde: uid = 350000393780939e4 (udev) Nov 04 18:59:46 | sde: detect_prio = 1 (config file default) Nov 04 18:59:46 | sde: prio = const (internal default) Nov 04 18:59:46 | sde: prio = (internal default) Nov 04 18:59:46 | sde: const prio = 1 Nov 04 18:59:46 | sdf: not found in pathvec Nov 04 18:59:46 | sdf: mask = 0x3f Nov 04 18:59:46 | sdf: dev_t = 8:80 Nov 04 18:59:46 | sdf: size = 1172123568 Nov 04 18:59:46 | sdf: vendor = IBM Nov 04 18:59:46 | sdf: product = MBF2600RC Nov 04 18:59:46 | sdf: rev = 610A Nov 04 18:59:46 | sdf: h:b:t:l = 1:0:7:0 Nov 04 18:59:46 | sdf: path state = running Nov 04 18:59:46 | sdf: 24018 cyl, 128 heads, 32 sectors/track, start at 0 Nov 04 18:59:46 | sdf: serial = EA09PB809C52 Nov 04 18:59:46 | sdf: get_state Nov 04 18:59:46 | sdf: path checker = directio (internal default) Nov 04 18:59:46 | sdf: checker timeout = 30000 ms (sysfs setting) Nov 04 18:59:46 | directio: starting new request Nov 04 18:59:46 | directio: io finished 4096/0 Nov 04 18:59:46 | sdf: state = up Nov 04 18:59:46 | sdf: uid_attribute = ID_SERIAL (internal default) Nov 04 18:59:46 | sdf: uid = 35000039378093b9c (udev) Nov 04 18:59:46 | sdf: detect_prio = 1 (config file default) Nov 04 18:59:46 | sdf: prio = const (internal default) Nov 04 18:59:46 | sdf: prio = (internal default) Nov 04 18:59:46 | sdf: const prio = 1 Nov 04 18:59:46 | sr0: device node name blacklisted Nov 04 18:59:46 | dm-0: device node name blacklisted Nov 04 18:59:46 | dm-1: device node name blacklisted Nov 04 18:59:46 | dm-10: device node name blacklisted Nov 04 18:59:46 | dm-2: device node name blacklisted Nov 04 18:59:46 | dm-3: device node name blacklisted Nov 04 18:59:46 | dm-4: device node name blacklisted Nov 04 18:59:46 | dm-5: device node name blacklisted Nov 04 18:59:46 | dm-6: device node name blacklisted Nov 04 18:59:46 | dm-7: device node name blacklisted Nov 04 18:59:46 | dm-8: device node name blacklisted Nov 04 18:59:46 | dm-9: device node name blacklisted ===== paths list ===== uuid hcil dev dev_t pri dm_st chk_st vend/prod/rev 3500003937809e810 0:0:4:0 sda 8:0 1 undef ready IBM ,MBF2600RC 350000393780939e4 0:0:5:0 sdb 8:16 1 undef ready IBM ,MBF2600RC 35000039378093b9c 0:0:7:0 sdc 8:32 1 undef ready IBM ,MBF2600RC 3500003937809e810 1:0:4:0 sdd 8:48 1 undef ready IBM ,MBF2600RC 350000393780939e4 1:0:5:0 sde 8:64 1 undef ready IBM ,MBF2600RC 35000039378093b9c 1:0:7:0 sdf 8:80 1 undef ready IBM ,MBF2600RC Nov 04 18:59:46 | params = 0 0 2 1 service-time 0 1 2 8:32 1 1 service-time 0 1 2 8:80 1 1 Nov 04 18:59:46 | status = 2 0 0 0 2 1 A 0 1 2 8:32 A 0 0 1 E 0 1 2 8:80 A 0 0 1 Nov 04 18:59:46 | mpathc: disassemble map [0 0 2 1 service-time 0 1 2 8:32 1 1 service-time 0 1 2 8:80 1 1 ] Nov 04 18:59:46 | mpathc: disassemble status [2 0 0 0 2 1 A 0 1 2 8:32 A 0 0 1 E 0 1 2 8:80 A 0 0 1 ] Nov 04 18:59:46 | params = 0 0 2 1 service-time 0 1 2 8:16 1 1 service-time 0 1 2 8:64 1 1 Nov 04 18:59:46 | status = 2 0 0 0 2 1 A 0 1 2 8:16 A 0 0 1 E 0 1 2 8:64 A 0 0 1 Nov 04 18:59:46 | mpathb: disassemble map [0 0 2 1 service-time 0 1 2 8:16 1 1 service-time 0 1 2 8:64 1 1 ] Nov 04 18:59:46 | mpathb: disassemble status [2 0 0 0 2 1 A 0 1 2 8:16 A 0 0 1 E 0 1 2 8:64 A 0 0 1 ] Nov 04 18:59:46 | params = 0 0 2 1 service-time 0 1 2 8:0 1 1 service-time 0 1 2 8:48 1 1 Nov 04 18:59:46 | status = 2 0 0 0 2 1 A 0 1 2 8:0 A 0 0 1 E 0 1 2 8:48 A 0 0 1 Nov 04 18:59:46 | mpatha: disassemble map [0 0 2 1 service-time 0 1 2 8:0 1 1 service-time 0 1 2 8:48 1 1 ] Nov 04 18:59:46 | mpatha: disassemble status [2 0 0 0 2 1 A 0 1 2 8:0 A 0 0 1 E 0 1 2 8:48 A 0 0 1 ] Nov 04 18:59:46 | directio checker refcount 6 Nov 04 18:59:46 | const prioritizer refcount 6 Nov 04 18:59:46 | directio checker refcount 5 Nov 04 18:59:46 | const prioritizer refcount 5 Nov 04 18:59:46 | directio checker refcount 4 Nov 04 18:59:46 | const prioritizer refcount 4 Nov 04 18:59:46 | directio checker refcount 3 Nov 04 18:59:46 | const prioritizer refcount 3 Nov 04 18:59:46 | directio checker refcount 2 Nov 04 18:59:46 | const prioritizer refcount 2 Nov 04 18:59:46 | directio checker refcount 1 Nov 04 18:59:46 | const prioritizer refcount 1 Nov 04 18:59:46 | unloading const prioritizer Nov 04 18:59:46 | unloading directio checker [root@junostac4 ~]# dmsetup ls --target multipath mpathc (253, 2) mpathb (253, 4) mpatha (253, 0) could you mount /boot by UUID, so that you didn't need to specify a specific device? Possibly. Isn't /etc/multipath/bindings supposed to guarantee the same disk gets the same name every time? Are there some more debugging commands that you want me to run? internal RTT agreed to test multipath on x86 fedora, we should have their results till end of the week BTW [root@junostac4 ~]# sed -i -e 's,/dev/mapper/mpatha2 *,'`blkid -p -s UUID /dev/mapper/mpathb2 | cut -f2 -d' '`' ,' /etc/fstab does work. This looks like: UUID=b8ac4471-9641-48c8-bfcb-bc4c3ae26d42 /boot ext4 defaults 1 2 Oh, and how to reproduce this: I selected three multipath drives on the Installation Destination page. Then I chose automatic partitioning. Then I reclaimed all the content on the drives. This creates an LVM partition that spans three multipath drives. /boot will exist outside of the LVM partition on the first drive. And a prepboot partition will also be created. (In reply to Ben Marzinski from comment #11) > could you mount /boot by UUID, so that you didn't need to specify a specific > device? Isn't there an obvious bug here in that /etc/multipath/bindings is being ignored? When you boot up, and see the multipath devices name not matching what's in /etc/multipath/bindings. Could you check /var/log/messages. Shortly after multipathd starts up, you should see messages like: multipathd: <wwid>: unable to rename <old_binding> to <new_binding> (<new_binding> in use by <other_wwid>) On my machine, they look like: Nov 5 06:10:01 ask-07 multipathd: 3600d02300069c9ce09d41c4ac9c53200: unable to rename mpatha to mpathe (mpathe is used by 3600d0230000000000e13955cc3757803) Nov 5 06:10:01 ask-07 multipathd: 3600d0230000000000e13955cc3757800: unable to rename mpathb to mpathd (mpathd is used by 3600d0230000000000e13955cc3757802) Nov 5 06:10:01 ask-07 multipathd: mpathc: load table [0 488118272 multipath 0 0 2 1 service-time 0 1 1 8:48 1 service-time 0 1 1 8:128 1] Nov 5 06:10:01 ask-07 multipathd: 3600d0230000000000e13955cc3757802: unable to rename mpathd to mpathb (mpathb is used by 3600d0230000000000e13955cc3757800) Nov 5 06:10:01 ask-07 multipathd: 3600d0230000000000e13955cc3757803: unable to rename mpathe to mpatha (mpatha is used by 3600d02300069c9ce09d41c4ac9c53200) These will happen if the copy of /etc/multipath/bindings in the initramfs doesn't match the copy in the regular filesystem. Even if you don't see these, it would be helpful to look at /var/log/messages anyway. Created attachment 954164 [details]
/var/log/messages from failed system boot up
I don't see any "unable" in /var/log/messages. Do all of those multipath messages look correct, Ben? The /var/log/messages you attached appears to contain the messages from two booting attempts. The first one hits Nov 5 14:33:08 junostac4 systemd: Timed out waiting for device dev-mapper-mpatha2.device. And the second one appears to succeed. In both of them the messages look normal. There doesn't appear to be any attempt to rename the devices. The only real difference is that in the failing attempt, only one path for each device exists in early boot, and the second one is added after pivoting to the real fileystem. However, it appears that the second path gets added correctly. The results look different from in your earlier Comments. In Comment 10, where the multipath setup matches the bindings file, mpatha gets sda, mpathb gets sdb, and mpathc gets sdc. In Comment 8, where the multipath setup doesn't match, mpatha gets sdc, mpathb gets sdb, and mpathc gets sda. In both the two bootups in the posted /var/log/messages, mpatha gets sda, mpathb gets sdb and mpathc gets sdc. I realize that sda, sdb, and sdc don't always refer to the same device, but in cases like this they are generally pretty consistent. Unfortunately, none of the messages state the WWID, so it's impossible to verify which actual LUN got put where. So, what exactly am I looking at here? Is this an example of two bootups with no changes where one happens to fail, and the next happens to succeed. Or did you change the fstab after boot one, and so in boot two, it's looking in the correct place? Which leads me to another question. Is this a case where the user friendly name that multipath assigns changes from boot to boot, or does it does it always come up the same when you boot, and the only change is between what happened when you isntalled, and what happens on the subsequent boots? The two differences you see are due to two different installations. I have even seen installation work once. Anaconda will always mount /boot as /dev/mapper/mpatha2. It seems that, after installation, mpatha will be one of the three drives chosen randomly. Subsequent reboots with the same installation will be consistent. So, on an installation where mpatha doesn't agree with anaconda, can you show me the results of # cat /etc/multipath/bindings # lsinitrd -f /etc/multipath/bindings [anaconda root@junostac4 ~]# cat /etc/multipath/bindings # Multipath bindings, Version : 1.0 # NOTE: this file is automatically maintained by the multipath program. # You should not need to edit this file in normal circumstances. # # Format: # alias wwid # mpatha 3500003937809e810 mpathb 350000393780939e4 mpathc 35000039378093b9c [anaconda root@junostac4 ~]# lsinitrd -f /etc/multipath/bindings -bash: lsinitrd: command not found [anaconda root@junostac4 ~]# scp root.stglabs.ibm.com:/usr/bin/lsinitrd /usr/bin/ [anaconda root@junostac4 ~]# lsinitrd -f /etc/multipath/bindings [anaconda root@junostac4 ~]# bash -x lsinitrd -f /etc/multipath/bindings ... + zcat -- /boot/initramfs-3.17.1-302.fc21.ppc64.img + cpio --extract --verbose --quiet --to-stdout etc/multipath/bindings ... [anaconda root@junostac4 ~]# zcat -- /boot/initramfs-3.17.1-302.fc21.ppc64.img | cpio --list /etc/multipath/\* /etc/multipath/bindings /etc/multipath/wwids 764806 blocks [anaconda root@junostac4 ~]# zcat -- /boot/initramfs-3.17.1-302.fc21.ppc64.img | cpio --extract --verbose --quiet --to-stdout /etc/multipath/bindings 2>/dev/null # Multipath bindings, Version : 1.0 # NOTE: this file is automatically maintained by the multipath program. # You should not need to edit this file in normal circumstances. # # Format: # alias wwid # mpatha 350000393780939e4 mpathb 35000039378093b9c mpathc 3500003937809e810 [anaconda root@junostac4 ~]# ls -l /dev/disk/by-id/ total 0 lrwxrwxrwx. 1 root root 10 Nov 11 13:32 dm-name-live-base -> ../../dm-9 lrwxrwxrwx. 1 root root 10 Nov 11 13:32 dm-name-live-rw -> ../../dm-8 lrwxrwxrwx. 1 root root 10 Nov 11 13:32 dm-name-mpatha -> ../../dm-0 lrwxrwxrwx. 1 root root 10 Nov 11 13:32 dm-name-mpatha1 -> ../../dm-4 lrwxrwxrwx. 1 root root 10 Nov 11 13:32 dm-name-mpatha2 -> ../../dm-6 lrwxrwxrwx. 1 root root 10 Nov 11 13:32 dm-name-mpatha3 -> ../../dm-7 lrwxrwxrwx. 1 root root 10 Nov 11 13:32 dm-name-mpathb -> ../../dm-1 lrwxrwxrwx. 1 root root 10 Nov 11 13:32 dm-name-mpathb1 -> ../../dm-3 lrwxrwxrwx. 1 root root 10 Nov 11 13:32 dm-name-mpathc -> ../../dm-2 lrwxrwxrwx. 1 root root 10 Nov 11 13:32 dm-name-mpathc1 -> ../../dm-5 lrwxrwxrwx. 1 root root 10 Nov 11 13:32 dm-uuid-mpath-350000393780939e4 -> ../../dm-0 lrwxrwxrwx. 1 root root 10 Nov 11 13:32 dm-uuid-mpath-35000039378093b9c -> ../../dm-1 lrwxrwxrwx. 1 root root 10 Nov 11 13:32 dm-uuid-mpath-3500003937809e810 -> ../../dm-2 lrwxrwxrwx. 1 root root 10 Nov 11 13:32 dm-uuid-part1-mpath-350000393780939e4 -> ../../dm-4 lrwxrwxrwx. 1 root root 10 Nov 11 13:32 dm-uuid-part1-mpath-35000039378093b9c -> ../../dm-3 lrwxrwxrwx. 1 root root 10 Nov 11 13:32 dm-uuid-part1-mpath-3500003937809e810 -> ../../dm-5 lrwxrwxrwx. 1 root root 10 Nov 11 13:32 dm-uuid-part2-mpath-350000393780939e4 -> ../../dm-6 lrwxrwxrwx. 1 root root 10 Nov 11 13:32 dm-uuid-part3-mpath-350000393780939e4 -> ../../dm-7 lrwxrwxrwx. 1 root root 10 Nov 11 13:32 lvm-pv-uuid-AXVn0I-AVA3-AXpe-AFyx-z5x2-Lwye-ILi3yv -> ../../dm-7 lrwxrwxrwx. 1 root root 10 Nov 11 13:32 lvm-pv-uuid-gi0Uv6-tAoP-o7gC-tPdy-MOJt-8RqY-9913sM -> ../../dm-3 lrwxrwxrwx. 1 root root 10 Nov 11 13:32 lvm-pv-uuid-p4Wrml-gwK5-Lyib-mF42-OoOW-fsZF-crIGe3 -> ../../dm-5 lrwxrwxrwx. 1 root root 9 Nov 11 13:32 scsi-350000393780939e4 -> ../../sde lrwxrwxrwx. 1 root root 10 Nov 11 13:32 scsi-350000393780939e4-part1 -> ../../sde1 lrwxrwxrwx. 1 root root 10 Nov 11 13:32 scsi-350000393780939e4-part2 -> ../../sde2 lrwxrwxrwx. 1 root root 10 Nov 11 13:32 scsi-350000393780939e4-part3 -> ../../sde3 lrwxrwxrwx. 1 root root 9 Nov 11 13:32 scsi-35000039378093b9c -> ../../sdf lrwxrwxrwx. 1 root root 10 Nov 11 13:32 scsi-35000039378093b9c-part1 -> ../../sdf1 lrwxrwxrwx. 1 root root 9 Nov 11 13:32 scsi-3500003937809e810 -> ../../sdd lrwxrwxrwx. 1 root root 10 Nov 11 13:32 scsi-3500003937809e810-part1 -> ../../sdd1 lrwxrwxrwx. 1 root root 9 Nov 11 13:32 wwn-0x50000393780939e4 -> ../../sde lrwxrwxrwx. 1 root root 10 Nov 11 13:32 wwn-0x50000393780939e4-part1 -> ../../sde1 lrwxrwxrwx. 1 root root 10 Nov 11 13:32 wwn-0x50000393780939e4-part2 -> ../../sde2 lrwxrwxrwx. 1 root root 10 Nov 11 13:32 wwn-0x50000393780939e4-part3 -> ../../sde3 lrwxrwxrwx. 1 root root 9 Nov 11 13:32 wwn-0x5000039378093b9c -> ../../sdf lrwxrwxrwx. 1 root root 10 Nov 11 13:32 wwn-0x5000039378093b9c-part1 -> ../../sdf1 lrwxrwxrwx. 1 root root 9 Nov 11 13:32 wwn-0x500003937809e810 -> ../../sdd lrwxrwxrwx. 1 root root 10 Nov 11 13:32 wwn-0x500003937809e810-part1 -> ../../sdd1 [anaconda root@junostac4 ~]# multipath -l mpathc (3500003937809e810) dm-2 IBM ,MBF2600RC size=559G features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=0 status=active | `- 0:0:4:0 sda 8:0 active undef running `-+- policy='service-time 0' prio=0 status=enabled `- 1:0:4:0 sdd 8:48 active undef running mpathb (35000039378093b9c) dm-1 IBM ,MBF2600RC size=559G features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=0 status=active | `- 0:0:7:0 sdc 8:32 active undef running `-+- policy='service-time 0' prio=0 status=enabled `- 1:0:7:0 sdf 8:80 active undef running mpatha (350000393780939e4) dm-0 IBM ,MBF2600RC size=559G features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=0 status=active | `- 0:0:5:0 sdb 8:16 active undef running `-+- policy='service-time 0' prio=0 status=enabled `- 1:0:5:0 sde 8:64 active undef running So do I understand this situation correctly right now during the booted installer? /dev/sda is wwid 3500003937809e810 and is also /dev/mapper/mpathc and is also dm-2. That is directly conflicting with the data in /etc/multipath/bindings. So yes, it appears that the scsi device wwids do correctly match the multipath device wwids, but the alias is based on the initramfs /etc/multipath/bindings, which doesn't match the real root filesystm /etc/multipath/bindings From Comment 23: The real root filesystem version of /etc/multipath/bindings has: mpatha 3500003937809e810 mpathb 350000393780939e4 mpathc 35000039378093b9c It initramfs version of /etc/multipath/bindings has: mpatha 350000393780939e4 mpathb 35000039378093b9c mpathc 3500003937809e810 And your multipath -l output has: mpatha (350000393780939e4) mpathb (35000039378093b9c) mpathc (3500003937809e810) Now the question is, why don't the two versions of /etc/multipath/bindings match? David, how do these get created? I assumed that multipath creates the real filesystem /etc/multipath/bindings when it discovers the devices. Then at some point in the installation, dracut is run to create the initramfs, and it should just be copying the real filesystem /etc/multipath/bindings into the initramfs. But the initramfs copy of this file is different. Any idea how that could happen? I should point a bit of multipath stupidness that may be relevant here. Bug 1086825 deals with a case that is similar. In this case it appears that all the paths are going to the correct maps, but the that bug shows how two different bindings files could be created in the installer. When multipath first creates the devices, it will add the bindings to /etc/multipath/bindings. If that file is later deleted or moved and then multipath is run with -r or multipathd is restarted or reloaded, it will check to see if the devices have been renamed. When it does this, it will not find any bindings (because the file is gone), and it will create new ones. These won't necessarily match the devices' current names. I'm not sure how /etc/multipath/bindings could disappear while the installer is running, but if it did, you could end up with a different version. Starting in f22, multipath is smart enough that when it finds that an existing multipath device with a user_friendly_name doesn't have a binding in /etc/multipath/bindings, it will first try to register the existing name. This means that even if the bindings file disappeared during install, the file the multipath would create would be identical. However, for this problem to come up, /etc/multipath/bindings needs to get removed sometime during the installer run, so that one version can end up in the initramfs and one version can end up in the real root filesystem. And possibly, the actual problem here is completely unrelated to Bug 1086825. It's just a thought. multipathd creates the bindings file. blivet only copies it into the new rootfs at the end of the installation. AFAICT anaconda initiates the copy of the bindings file well before we recreate the initramfs. The only place anything like what you describe in comment 27 could happen is during the switchroot from the initramfs to the installer's runtime, but that would mean that multipathd is writing out a bogus bindings file in the installer runtime after the switchroot. multipathd runs (I think) in the initramfs, and then it gets somehow reloaded after we switchroot into the installer's real runtime evironment. Ben, I remember there being some fix required to multipathd to accomplish this without losing track of what was done in the initramfs. Sound familiar? To be clear, neither anaconda nor blivet moves or removes the bindings file. Blivet copies it from the installer runtime into the installed system. Has the f22 version of multipath that you are talking about been built yet? Is there a way to test that behavior now? Yeah: http://kojipkgs.fedoraproject.org/packages/device-mapper-multipath/0.4.9/68.fc22/ I can build an updates.img file with the new multipath packages. You'd just copy the image file to: http://some.website.com/path/to/updates.img and use the: inst.updates=http://some.website.com/path/to/updates.img kernel command line when booting into the installer. It's also possible to put the updates.img in images/ directory of your installation tree for CD, hard drive, HTTP, NFS or FTP installs. For NFS installs, you can also explode the image into the RHupdates/ directory of the installation tree. The updates.img will overrwrite the device-mapper-multipath files that are used in the installation process. It shouldn't matter the version of the tools you use in the installation process doesn't match the version that the installed system uses. I don't see the bindings switch at all when causing multipath to reload: [anaconda root@junostac4 ~]# cat /etc/multipath/bindings | sed -e '/^#/d' mpatha 3500003937809e810 mpathb 350000393780939e4 mpathc 35000039378093b9c [anaconda root@junostac4 ~]# dmsetup remove_all [anaconda root@junostac4 ~]# dmsetup -v ls live-rw (253:8) [anaconda root@junostac4 ~]# rm /etc/multipath/bindings /etc/multipath/wwids [anaconda root@junostac4 ~]# multipath -v3 [anaconda root@junostac4 ~]# cat /etc/multipath/bindings | sed -e '/^#/d' mpatha 3500003937809e810 mpathb 350000393780939e4 mpathc 35000039378093b9c Am I doing that right? I can run an updates.img file easily enough. Although it would need to be build for ppc64le. Dan is performing a scratch build of http://ppc.koji.fedoraproject.org/koji/taskinfo?taskID=2178471 (In reply to David Lehman from comment #28) > multipathd creates the bindings file. blivet only copies it into the new > rootfs at the end of the installation. AFAICT anaconda initiates the copy of > the bindings file well before we recreate the initramfs. > > The only place anything like what you describe in comment 27 could happen is > during the switchroot from the initramfs to the installer's runtime, but > that would mean that multipathd is writing out a bogus bindings file in the > installer runtime after the switchroot. multipathd runs (I think) in the > initramfs, and then it gets somehow reloaded after we switchroot into the > installer's real runtime evironment. Ben, I remember there being some fix > required to multipathd to accomplish this without losing track of what was > done in the initramfs. Sound familiar? Yeah. Multipathd needed the register the wwids of the already existing paths with /etc/multipath/wwids. It actually needed to register the already existing bindings with /etc/multipath/bindings to, but it didn't. That's what Bug 1086825 fixes. I don't think that 1086825 happening during during the installer would get us this outcome. I could see that causing the wrong bindings file to end up in both the initramfs and real root filesystem. But in this case dracut copied the correct version of the file into the initramfs. Multipath has no code in it to delete either the entire bindings file or any part of it. It can only add new lines. But looking at the results, that file did get changed between when it was copied to the real root filesystem and the initramfs. One thing that makes no sense to me, however, is the order. If anaconda copies the file the real root filesystem first, and dracut makes the initramfs second, that means that when anaconda runs, the file is incorrect, but when dracut runs later, the file is correct. > To be clear, neither anaconda nor blivet moves or removes the bindings file. > Blivet copies it from the installer runtime into the installed system. (In reply to Mark Hamzy from comment #31) > I don't see the bindings switch at all when causing multipath to reload: > > [anaconda root@junostac4 ~]# cat /etc/multipath/bindings | sed -e '/^#/d' > mpatha 3500003937809e810 > mpathb 350000393780939e4 > mpathc 35000039378093b9c > [anaconda root@junostac4 ~]# dmsetup remove_all > [anaconda root@junostac4 ~]# dmsetup -v ls > live-rw (253:8) > [anaconda root@junostac4 ~]# rm /etc/multipath/bindings /etc/multipath/wwids > [anaconda root@junostac4 ~]# multipath -v3 > [anaconda root@junostac4 ~]# cat /etc/multipath/bindings | sed -e '/^#/d' > mpatha 3500003937809e810 > mpathb 350000393780939e4 > mpathc 35000039378093b9c > > > Am I doing that right? > > I can run an updates.img file easily enough. Although it would need to be > build for ppc64le. Dan is performing a scratch build of > http://ppc.koji.fedoraproject.org/koji/taskinfo?taskID=2178471 It won't change the bindings file. multipath can't. It can only add to it. But it should make the the multipath devices match the bindings file. http://fedora.danny.cz/ppc/updates.img from the above build for ppc64 The updates.img file seems to fix the issue. Although it does not get installed as this method is supposed to only fix the installer runtime. So, how much code is different in device-mapper-multipath-0.4.9-68.fc22? And can you build that release in fc21? And with a new version number I assume? (In reply to Mark Hamzy from comment #35) > The updates.img file seems to fix the issue. Although it does not get > installed as this method is supposed to only fix the installer runtime. Great. Since we're in the post beta development freeze, I can't pull all the changes back. There are a number of unrelated feature enhancements and in post beta, we can only add patches to fix bugs. But I can pull back the releated changes (there are actually 3 associated patches). > So, how much code is different in device-mapper-multipath-0.4.9-68.fc22? And > can you build that release in fc21? And with a new version number I assume? This is actually the second post beta freeze request for a backported fix to f21. Since according to rpm: 0.4.9-68.fc21 < 0.4.9-68.fc22 I just backported the patch and bumped the version. In hindsight, I probably should have used -0.4.9-67.fc21.1 like rhel does for zstream releases. Technically: 0.4.9-68.fc21.1 < 0.4.9-68.fc22 But that looks much more like the package got downgraded, so I'll probably also rebuild the f22 release, just to bump the version number to -69, so they look like 0.4.9-68.fc21.1 and 0.4.9-69.fc22 Lessons learned. device-mapper-multipath-0.4.9-68.fc21.1 has been submitted as an update for Fedora 21. https://admin.fedoraproject.org/updates/device-mapper-multipath-0.4.9-68.fc21.1 device-mapper-multipath-0.4.9-68.fc21.1 has been built with the rename patches backported. Like I mentioned, it doesn't have everything that the fc22 build did, so it would be worthwhile retesting with this. It's currently in testing. updates image build from the device-mapper-multipath-0.4.9-68.fc21.1 rpms http://fedora.danny.cz/ppc/updates-new.img Proposed as a Blocker for 21-final by Fedora user sharkcz using the blocker tracking app because: Installs on FC multipath devices are broken and multipath installation is a criteria for final. Package device-mapper-multipath-0.4.9-68.fc21.1: * should fix your issue, * was pushed to the Fedora 21 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing device-mapper-multipath-0.4.9-68.fc21.1' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2014-15126/device-mapper-multipath-0.4.9-68.fc21.1 then log in and leave karma (feedback). device-mapper-multipath-0.4.9-68.fc21.1 has been pushed to the Fedora 21 stable repository. If problems still persist, please make note of it in this bug report. |