Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1150147

Summary: Parent of existing mdcontainer does not have mdmember format - "ValueError: member has wrong format" on Intel firmware RAID install of F21
Product: [Fedora] Fedora Reporter: mulhern <amulhern>
Component: python-blivetAssignee: mulhern <amulhern>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 21CC: amulhern, anaconda-maint-list, awilliam, bcl, dlehman, extras-qa, g.kaviyarasu, jbowm16, jonathan, jrimpo, robatino, vanmeeuwen+fedora, vpodzime
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard: AcceptedBlocker
Fixed In Version: anaconda-21.48.10-1.fc21 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1145783
: 1151649 (view as bug list) Environment:
Last Closed: 2014-10-20 23:02:06 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1145783    
Bug Blocks: 1043124    
Attachments:
Description Flags
Relevant tf file from cloned bug
none
Error log from anaconda/blivet 0.61.5
none
AttributeError: 'MDRaidArrayDevice' object has no attribute '_partedDevice' none

Description mulhern 2014-10-07 14:08:33 UTC
--- Additional comment from Jeremy Rimpo on 2014-10-02 15:56:36 EDT ---

After some trial and error, I got the img to apply and made some progress. Whereas before anaconda crashed before even selecting language, I can now make it slightly past that point before a new error is triggered.

Attaching the contents of the anaconda error report.

--- Additional comment from Vratislav Podzimek on 2014-10-02 16:12:18 EDT ---

(In reply to Jeremy Rimpo from comment #29)
> Created attachment 943543 [details]
> Anaconda error log 21 alpha 1 - applying updates.img #1
> 
> After some trial and error, I got the img to apply and made some progress.
> Whereas before anaconda crashed before even selecting language, I can now
> make it slightly past that point before a new error is triggered.
> 
> Attaching the contents of the anaconda error report.
Interesting::
Traceback (most recent call first):
  File "/tmp/updates/blivet/devices.py", line 2347, in _addParent
    raise ValueError("member has wrong format")
...


Local variables in innermost frame:
member: existing 1863.03 GiB mdcontainer imsm0 (20)

--- Additional comment from Jeremy Rimpo on 2014-10-06 10:50:51 EDT ---

This may or may not be related to the newer error, but it may help debug in the off chance. My RAID drive is an Intel Software Raid of two 1TB drives (as you most likely already noticed) which is further partitioned into one large data portion (I think 1.6 TB) as NTFS and the rest is used by Fedora as a mix of Ext4 as root and and LVM2 as home data. Using MBR partitioning.

--- Additional comment from Jeremy Rimpo on 2014-10-06 11:12:21 EDT ---

Correction, there's a Ext4 boot partition, and the LVM is a mix of root, home, and swap partitions.

Comment 1 mulhern 2014-10-07 14:09:36 UTC
Created attachment 944607 [details]
Relevant tf file from cloned bug

Comment 2 Adam Williamson 2014-10-08 22:01:36 UTC
Nominating as a 21 Beta Blocker: https://fedoraproject.org/wiki/Fedora_21_Beta_Release_Criteria#Hardware_and_firmware_RAID , "The installer must be able to detect and install to hardware or firmware RAID storage devices." I have reproduced this on my Intel firmware RAID test array.

Comment 3 Jeremy Rimpo 2014-10-10 14:55:01 UTC
Is there any way to test this before a new install image is generated?

Comment 4 mulhern 2014-10-10 16:04:30 UTC
Back at drawing board.

Comment 5 mulhern 2014-10-10 20:41:40 UTC
(In reply to Jeremy Rimpo from comment #3)
> Is there any way to test this before a new install image is generated?

You can make your own updates.img using scripts/makeupdates in blivet directory.

Comment 6 mulhern 2014-10-10 21:20:58 UTC
Also pushed to master.

Comment 7 Fedora Update System 2014-10-15 13:15:45 UTC
anaconda-21.48.10-1.fc21, pykickstart-1.99.63-2.fc21, python-blivet-0.61.5-1.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/pykickstart-1.99.63-2.fc21,python-blivet-0.61.5-1.fc21,anaconda-21.48.10-1.fc21

Comment 8 Adam Williamson 2014-10-15 16:19:34 UTC
Discussed at 2014-10-15 blocker review meeting: http://meetbot.fedoraproject.org/fedora-blocker-review/2014-10-15/f21-blocker-review.2014-10-15-16.04.log.txt . Accepted as a blocker per criterion cited in c#2.

Comment 9 Fedora Update System 2014-10-16 17:18:43 UTC
Package anaconda-21.48.10-1.fc21, pykickstart-1.99.63-2.fc21, python-blivet-0.61.5-1.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing anaconda-21.48.10-1.fc21 pykickstart-1.99.63-2.fc21 python-blivet-0.61.5-1.fc21'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-12944/pykickstart-1.99.63-2.fc21,python-blivet-0.61.5-1.fc21,anaconda-21.48.10-1.fc21
then log in and leave karma (feedback).

Comment 10 Jeremy Rimpo 2014-10-16 21:07:00 UTC
Created attachment 947747 [details]
Error log from anaconda/blivet 0.61.5

Unfortunately, I'm now being greeted with yet another error. I tried both updating the version of anaconda and python-blivet as well as generating an updates.img from the current source release. The result was the same.

I will provide full error report output, if needed.

Comment 11 mulhern 2014-10-16 21:27:59 UTC
If stack trace looks like this one

anaconda 19.31.100-1 exception report
Traceback (most recent call first):
  File "/tmp/updates/blivet/devices.py", line 2440, in _addParent
    getattr(member.format, self._formatUUIDAttr) != self.uuid:
  File "/tmp/updates/blivet/devices.py", line 3964, in _addParent
    super(MDRaidArrayDevice, self)._addParent(member)
  File "/tmp/updates/blivet/devices.py", line 181, in append
    self.appendfunc(y)
  File "/tmp/updates/blivet/devices.py", line 323, in _setParentList
    self._parents.append(parent)
  File "/tmp/updates/blivet/devices.py", line 266, in __init__
    self.parents = parents
  File "/tmp/updates/blivet/devices.py", line 568, in __init__
    Device.__init__(self, name, parents=parents)
  File "/tmp/updates/blivet/devices.py", line 2410, in __init__
    super(ContainerDevice, self).__init__(*args, **kwargs)
  File "/tmp/updates/blivet/devices.py", line 3745, in __init__
    sysfsPath=sysfsPath)
  File "/tmp/updates/blivet/devicetree.py", line 1060, in addUdevDiskDevice
    sysfsPath=sysfs_path, **kwargs)
  File "/tmp/updates/blivet/devicetree.py", line 1193, in addUdevDevice
    device = self.addUdevDiskDevice(info)
  File "/tmp/updates/blivet/devicetree.py", line 921, in addUdevPartitionDevice
    self.addUdevDevice(new_info)
  File "/tmp/updates/blivet/devicetree.py", line 1196, in addUdevDevice
    device = self.addUdevPartitionDevice(info)
  File "/tmp/updates/blivet/devicetree.py", line 693, in addUdevLVDevice
    self.addUdevDevice(pv_info)
  File "/tmp/updates/blivet/devicetree.py", line 1175, in addUdevDevice
    device = self.addUdevLVDevice(info)
  File "/tmp/updates/blivet/devicetree.py", line 2169, in _populate
    self.addUdevDevice(dev)
  File "/tmp/updates/blivet/devicetree.py", line 2104, in populate
    self._populate()
  File "/tmp/updates/blivet/__init__.py", line 482, in reset
    self.devicetree.populate(cleanupOnly=cleanupOnly)
  File "/tmp/updates/blivet/__init__.py", line 186, in storageInitialize
    storage.reset()
  File "/usr/lib64/python2.7/threading.py", line 764, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib64/python2.7/site-packages/pyanaconda/threads.py", line 211, in run
    threading.Thread.run(self, *args, **kwargs)
AttributeError: 'DeviceFormat' object has no attribute 'mdUuid'

no need to tell me anymore, I should be pushing a patch in a bit.

Comment 12 Jeremy Rimpo 2014-10-16 21:34:34 UTC
Yep. That's the error. Only difference between update methods is the path to the python scripts.

Comment 13 mulhern 2014-10-16 21:42:10 UTC
And pushed.

Comment 14 Jeremy Rimpo 2014-10-16 22:35:04 UTC
Now something else:

Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/pyanaconda/threads.py", line 227, in run
    threading.Thread.run(self, *args, **kwargs)
  File "/usr/lib64/python2.7/threading.py", line 766, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib64/python2.7/site-packages/pyanaconda/packaging/__init__.py", line 1222, in _runThread
    threadMgr.wait(THREAD_STORAGE)
  File "/usr/lib64/python2.7/site-packages/pyanaconda/threads.py", line 112, in wait
    self.raise_if_error(name)
  File "/usr/lib64/python2.7/site-packages/pyanaconda/threads.py", line 227, in run
    threading.Thread.run(self, *args, **kwargs)
  File "/usr/lib64/python2.7/threading.py", line 766, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/tmp/updates/blivet/__init__.py", line 183, in storageInitialize
    storage.reset()
  File "/tmp/updates/blivet/__init__.py", line 479, in reset
    self.devicetree.populate(cleanupOnly=cleanupOnly)
  File "/tmp/updates/blivet/devicetree.py", line 2105, in populate
    self._populate()
  File "/tmp/updates/blivet/devicetree.py", line 2170, in _populate
    self.addUdevDevice(dev)
  File "/tmp/updates/blivet/devicetree.py", line 1180, in addUdevDevice
    device = self.addUdevLVDevice(info)
  File "/tmp/updates/blivet/devicetree.py", line 698, in addUdevLVDevice
    self.addUdevDevice(pv_info)
  File "/tmp/updates/blivet/devicetree.py", line 1201, in addUdevDevice
    device = self.addUdevPartitionDevice(info)
  File "/tmp/updates/blivet/devicetree.py", line 926, in addUdevPartitionDevice
    self.addUdevDevice(new_info)
  File "/tmp/updates/blivet/devicetree.py", line 1198, in addUdevDevice
    device = self.addUdevDiskDevice(info)
  File "/tmp/updates/blivet/devicetree.py", line 1065, in addUdevDiskDevice
    sysfsPath=sysfs_path, **kwargs)
  File "/tmp/updates/blivet/devices.py", line 3733, in __init__
    sysfsPath=sysfsPath)
  File "/tmp/updates/blivet/devices.py", line 2349, in __init__
    super(ContainerDevice, self).__init__(*args, **kwargs)
  File "/tmp/updates/blivet/devices.py", line 570, in __init__
    Device.__init__(self, name, parents=parents)
  File "/tmp/updates/blivet/devices.py", line 268, in __init__
    self.parents = parents
  File "/tmp/updates/blivet/devices.py", line 325, in _setParentList
    self._parents.append(parent)
  File "/tmp/updates/blivet/devices.py", line 183, in append
    self.appendfunc(y)
  File "/tmp/updates/blivet/devices.py", line 3968, in _addParent
    if self.status and member.format.exists:
  File "/tmp/updates/blivet/devices.py", line 4006, in status
    if os.path.exists(self.path) and not self.sysfsPath:
AttributeError: 'MDRaidArrayDevice' object has no attribute 'sysfsPath'

Comment 15 mulhern 2014-10-17 16:53:26 UTC
This is a superclass constructor initalization ordering bug.

We're checking the status of the array while adding its parents.

Ordering is:

_memberDevices = ...
_totalDevices = ...
ContainerDevice.__init__
    formatClass = ...
    StorageDevice.__init__
        exists = ...
        uuid = ...
        Device.__init__
            util.ObjectID = ...
            kids = ...
            _name = ...
            parents = ...   (invokes _addParent() which requires sysfsPath)
        sysfsPath = ... (too late!)

This is a bit funny, because _addParent() is being invoked during device discovery, and shouldn't really need to be doing something w/ the actual device.
But, there is a reason, the size needs to be discovered while the array is up, so the code takes every chance it gets.
            
Best thing to do is to reorder the two assignments.

Comment 16 Jeremy Rimpo 2014-10-17 17:33:21 UTC
Makes sense. As soon as I have a commit to pull, I'll give it another go. Hopefully it'll be the last time!

Comment 17 mulhern 2014-10-17 18:38:36 UTC
Hopefully...

Comment 18 Jeremy Rimpo 2014-10-17 19:39:43 UTC
Created attachment 948009 [details]
AttributeError: 'MDRaidArrayDevice' object has no attribute '_partedDevice'

Unfortunately, we've run into another one. It took a lot longer to hit this one. Seems to be another missing attribute.

Comment 19 mulhern 2014-10-17 20:56:31 UTC
One line further and about the same number of stack frames deep. Basically the same problem, however, and with, presumably, the same solution.

Could you try with updates image http://mulhern.fedorapeople.org/1150147.img and see if you get past this?

- mulhern

Comment 20 Jeremy Rimpo 2014-10-17 21:13:18 UTC
Tentative success!

I dug in and was able to correctly select/modify partitions from the RAID drive to set up the installation. There was some kind of atk-bridge/dbus warning on startup, but I think that is a totally separate issue. Do I need to run an actuall install to the drive to totally verify - or does it look good now?

Comment 21 mulhern 2014-10-17 21:41:20 UTC
(In reply to Jeremy Rimpo from comment #20)
> Tentative success!
> 
> I dug in and was able to correctly select/modify partitions from the RAID
> drive to set up the installation. There was some kind of atk-bridge/dbus
> warning on startup, but I think that is a totally separate issue. Do I need
> to run an actuall install to the drive to totally verify - or does it look
> good now?

That's good news.

One never knows what will happen...but I think that if you ran into a bug during the install that you would probably want to file it as a separate issue.

These problems all fell under the same category, "Unanticipated problems when instantiating ContainerDevices that happen to be MDRaidArrayDevices", and I think any subsequent problem would have to be something different.

Thanks for the quick turnaround.

- mulhern

Comment 22 Fedora Update System 2014-10-20 19:01:57 UTC
anaconda-21.48.11-1.fc21, python-blivet-0.61.6-1.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/python-blivet-0.61.6-1.fc21,anaconda-21.48.11-1.fc21

Comment 23 Fedora Update System 2014-10-20 23:02:06 UTC
anaconda-21.48.10-1.fc21, pykickstart-1.99.63-2.fc21, python-blivet-0.61.5-1.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 24 Fedora Update System 2014-10-23 14:15:07 UTC
anaconda-21.48.12-1.fc21, python-blivet-0.61.7-1.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/python-blivet-0.61.7-1.fc21,anaconda-21.48.12-1.fc21

Comment 25 Adam Williamson 2014-10-24 19:20:25 UTC
FWIW kparal and I are both now hitting https://bugzilla.redhat.com/show_bug.cgi?id=1156354 . dlehman has a fix for that, and with that fix the install finally completes for me, but on booting the installed system I hit https://bugzilla.redhat.com/show_bug.cgi?id=1156614 .