Bug 587627

Proposing as a F13Blocker.  Since running low on /boot disk space is common, this issue came up multiple times during the test day.  I'd like to propose moving the kickstart generation into it's own try/except block so that it can still be written to disk.

note that, as usual for preupgrade blockers, this blocks F13 release but not composes; as long as we can fix it before the actual planned release date, it's okay.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

(In reply to comment #1)
> Proposing as a F13Blocker.  Since running low on /boot disk space is common,
> this issue came up multiple times during the test day.  I'd like to propose
> moving the kickstart generation into it's own try/except block so that it can
> still be written to disk.    

You mentioned a patch on the test day, two questions:

1. could you attach it please
2. is it tested?

Thanks.

Created attachment 411323 [details]
preupgrade.patch - move kickstart generation before install.img download

I'm testing the attached patch as we speak.  This patch only fixes preupgrade, preupgrade-cli would need a similar change I gather.

(In reply to comment #3)
> (In reply to comment #1)
> > Proposing as a F13Blocker.  Since running low on /boot disk space is common,
> > this issue came up multiple times during the test day.  I'd like to propose
> > moving the kickstart generation into it's own try/except block so that it can
> > still be written to disk.    
> 
> You mentioned a patch on the test day, two questions:
> 
> 1. could you attach it please

See attachment#411323 [details]

> 2. is it tested?

Yes, I can confirm that this patch creates a kickstart file and allows for clean preupgrade in conditions where there is enough '/boot' space for vmlinuz+initrd.img, but *not* enough for install.img.

commit 03ccc131633b7c19e4d4fe11d689fdbfb47efb15
Author: James Laska <jlaska>
Date:   Wed May 5 09:17:42 2010 +0100

    Generate a valid kickstart when there is no space for the install.img
    
    Signed-off-by: Richard Hughes <richard>

Please provide a build so we can test it.

Can you test the build here please: http://people.freedesktop.org/~hughsient/fedora/13/i386/

It's got some other fixups that might be also good to test. Thanks.

Tested preupgrade-1.1.6-0.10.fc13.noarch.rpm:

# preupgrade
Loaded plugins: blacklist, whiteout
No plugin match for: rpm-warm-cache
No plugin match for: remove-with-leaves
No plugin match for: auto-update-debuginfo
preupgrade-main (mirrorlist) 
  url: http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-13&arch=$basearch
  now: http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-13&arch=x86_64
preupgrade (mirrorlist) 
  url: http://mirrors.fedoraproject.org/mirrorlist?path=pub/fedora/linux/development/13/$basearch/os
  now: http://mirrors.fedoraproject.org/mirrorlist?path=pub/fedora/linux/development/13/x86_64/os
unknown metadata being downloaded: repomd4SEA3itmp.xml
unknown metadata being downloaded: MEMORY
Fetched treeinfo from http://download.englab.brq.redhat.com/pub/fedora/linux/development/13/x86_64/os//.treeinfo
treeinfo timestamp: Wed May  5 09:37:07 2010
unknown metadata being downloaded: MEMORY
unknown metadata being downloaded: MEMORY
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 1082, in _retrieve
    self.fo.write(buf)
IOError: [Errno 28] No space left on device
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 1082, in _retrieve
    self.fo.write(buf)
IOError: [Errno 28] No space left on device
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 1082, in _retrieve
    self.fo.write(buf)
IOError: [Errno 28] No space left on device
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 1082, in _retrieve
    self.fo.write(buf)
IOError: [Errno 28] No space left on device
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 1082, in _retrieve
    self.fo.write(buf)
IOError: [Errno 28] No space left on device
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 1082, in _retrieve
    self.fo.write(buf)
IOError: [Errno 28] No space left on device
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 1082, in _retrieve
    self.fo.write(buf)
IOError: [Errno 28] No space left on device
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 1082, in _retrieve
    self.fo.write(buf)
IOError: [Errno 28] No space left on device
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 1082, in _retrieve
    self.fo.write(buf)
IOError: [Errno 28] No space left on device
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 1082, in _retrieve
    self.fo.write(buf)
IOError: [Errno 28] No space left on device
Traceback (most recent call last):
  File "/usr/share/preupgrade/preupgrade-gtk.py", line 259, in on_assistant_apply
    self._do_main()
  File "/usr/share/preupgrade/preupgrade-gtk.py", line 278, in _do_main
    self.main_preupgrade()
  File "/usr/share/preupgrade/preupgrade-gtk.py", line 500, in main_preupgrade
    stage2file = self.pu.retrieve_non_critical_files()
  File "/usr/lib/python2.6/site-packages/preupgrade/__init__.py", line 571, in retrieve_non_critical_files
    self._retrieve_file(self.mainimage, targetdir, reserve_space=extra_space)
  File "/usr/lib/python2.6/site-packages/preupgrade/__init__.py", line 479, in _retrieve_file
    self.instrepo._getFile(relative=fileinfo, local=local)
  File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 823, in _getFile
    size=size
  File "/usr/lib/python2.6/site-packages/urlgrabber/mirror.py", line 408, in urlgrab
    return self._mirror_try(func, url, kw)
  File "/usr/lib/python2.6/site-packages/urlgrabber/mirror.py", line 394, in _mirror_try
    return func_ref( *(fullurl,), **kwargs )
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 976, in urlgrab
    return self._retry(opts, retryfunc, url, filename)
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 880, in _retry
    r = apply(func, (opts,) + args, {})
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 962, in retryfunc
    fo = PyCurlFileObject(url, filename, opts)
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 1056, in __init__
    self._do_open()
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 1314, in _do_open
    self._do_grab()
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 1444, in _do_grab
    self._do_perform()
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 1244, in _do_perform
    raise KeyboardInterrupt
KeyboardInterrupt
[root@localhost ~]# 

The scary part is that preupgrade gui reports "Ready to upgrade, reboot now". That should never happen.

Kamil, does your F-13 have the fixed python-urlgrabber package? See bugs #587627 and #587627

I worked with jlaska today on this issue and ran some tests on a fresh install of Fedora 12 x86. I did an install selecting all of the defaults except in the area of partitioning. I changed the /boot partition from the default 200mb to 1024mb.

Once the system was fully updated to the latest F12 patch level I installed both http://people.freedesktop.org/~hughsient/fedora/13/i386/preupgrade-1.1.6-0.10.fc13.noarch.rpm and http://kojipkgs.fedoraproject.org/packages/python-urlgrabber/3.9.1/4.1.fc12/noarch/python-urlgrabber-3.9.1-4.1.fc12.noarch.rpm

After these two packages were updated I ran preupgrade as root. The process was very smooth after that. Following is the output that preupgrade generated:

[root@oneil ~]# preupgrade
Loaded plugins: blacklist, whiteout
No plugin match for: rpm-warm-cache
No plugin match for: remove-with-leaves
No plugin match for: auto-update-debuginfo
preupgrade-main (mirrorlist) 
  url: http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-13&arch=$basearch
  now: http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-13&arch=i386
preupgrade (mirrorlist) 
  url: http://mirrors.fedoraproject.org/mirrorlist?path=pub/fedora/linux/development/13/$basearch/os
  now: http://mirrors.fedoraproject.org/mirrorlist?path=pub/fedora/linux/development/13/i386/os
unknown metadata being downloaded: repomdG9KuvBtmp.xml
unknown metadata being downloaded: MEMORY
Fetched treeinfo from http://mirrordenver.fdcservers.net/fedora/development/13/i386/os//.treeinfo
treeinfo timestamp: Thu May  6 10:16:56 2010
unknown metadata being downloaded: MEMORY
unknown metadata being downloaded: MEMORY
Downloading 760.1MB
Available disk space for /var/cache/yum/preupgrade: 61.5GB
Upgrade requires 500.0MB
Available disk space for /usr: 60.8GB
Kernel requires 26.0MB
Available disk space for /boot: 716.8MB
unknown metadata being downloaded: repomdVrUGr2tmp.xml
Generating metadata for preupgrade repo
DEBUG /sbin/grubby --title="Upgrade to Fedora 13 (Branch)" --remove-kernel="/boot/upgrade/vmlinuz" --add-kernel="/boot/upgrade/vmlinuz" --initrd="/boot/upgrade/initrd.img" --args="preupgrade repo= hd::/var/cache/yum/preupgrade  ks= hd:UUID=54bb5aef-5b7a-4af8-b4c2-581ac7bc85cb:/upgrade/ks.cfg stage2= hd:UUID=54bb5aef-5b7a-4af8-b4c2-581ac7bc85cb:/upgrade/install.img"

jlaska asked what the grub.conf looked like once it was complete. Following is the /boot/grub/grub.conf that was generated:

# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/mapper/vg_oneil-LogVol00
#          initrd /initrd-[generic-]version.img
#boot=/dev/sda
default=1
timeout=0
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Upgrade to Fedora 13 (Branch)
	kernel /upgrade/vmlinuz preupgrade repo= hd::/var/cache/yum/preupgrade ks= hd:UUID=54bb5aef-5b7a-4af8-b4c2-581ac7bc85cb:/upgrade/ks.cfg stage2= hd:UUID=54bb5aef-5b7a-4af8-b4c2-581ac7bc85cb:/upgrade/install.img
	initrd /upgrade/initrd.img
title Fedora (2.6.32.11-99.fc12.i686.PAE)
	root (hd0,0)
	kernel /vmlinuz-2.6.32.11-99.fc12.i686.PAE ro root=/dev/mapper/vg_oneil-LogVol00  LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb quiet
	initrd /initramfs-2.6.32.11-99.fc12.i686.PAE.img
title Fedora (2.6.31.5-127.fc12.i686.PAE)
	root (hd0,0)
	kernel /vmlinuz-2.6.31.5-127.fc12.i686.PAE ro root=/dev/mapper/vg_oneil-LogVol00  LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb quiet
	initrd /initramfs-2.6.31.5-127.fc12.i686.PAE.img

The thing that seemed odd to jlaska and I was the spaces in the kernel line. For instance repo= hd::

Following his advice I edited the grub.conf file to add 

title Upgrade to Fedora 13 (Branch) -- MODIFIED
	kernel /upgrade/vmlinuz preupgrade repo=hd::/var/cache/yum/preupgrade ks=hd:UUID=54bb5aef-5b7a-4af8-b4c2-581ac7bc85cb:/upgrade/ks.cfg stage2=hd:UUID=54bb5aef-5b7a-4af8-b4c2-581ac7bc85cb:/upgrade/install.img
	initrd /upgrade/initrd.img

after the first F13 kernel line.

After this I rebooted the system and it booted to the F13 upgrade screens and it progressed as expected.

Ok, I updated to python-urlgrabber-3.9.1-4.1.fc12 and that solved that traceback. Kickstart is now generated when not enough space for install.img, good. But there are still some problems:

1. preupgrade does not require python-urlgrabber-3.9.1-4.1.fc12. It must require that, because you can't suppose the user has all packages fully updated.
2. Upgrade item in GRUB is not set as default. I have seen this for several times, it seems to appear randomly (after some preupgrade runs it is default, after some it isn't).
3. There are superfluous spaces in GRUB item, exactly as Eric Lake reported. That makes the whole process fail ("kickstart not found").

(In reply to comment #12)
> 2. Upgrade item in GRUB is not set as default. I have seen this for several
> times, it seems to appear randomly (after some preupgrade runs it is default,
> after some it isn't).
> 3. There are superfluous spaces in GRUB item, exactly as Eric Lake reported.
> That makes the whole process fail ("kickstart not found").    

Different bug reports please, please don't piggy back new things onto this ticket. Thanks.

preupgrade-1.1.6-1.fc12 has been submitted as an update for Fedora 12.
http://admin.fedoraproject.org/updates/preupgrade-1.1.6-1.fc12

I'm not piggybacking, those problems are directly connected to this update (except for #2).

Tested preupgrade-1.1.6-1.fc12, from comment #12 - problem 1 is solved, problem 2 has not appeared, but problem 3 is still present and that is a show-stopper.

preupgrade-1.1.6-1.fc12 has been pushed to the Fedora 12 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update preupgrade'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/preupgrade-1.1.6-1.fc12

(In reply to comment #12)
> 3. There are superfluous spaces in GRUB item, exactly as Eric Lake reported.
> That makes the whole process fail ("kickstart not found").    

I also ran into this issue with preupgrade-1.1.6-1.fc12.  Removing the spaces in /etc/grup.conf by hand seems to have allowed the upgrade to proceed.

I just want to add that I just had problems upgrading from F12 to F13 because of a *missing* space in grub.conf. I have a small /boot, and thus a http link was inserted as the stage2 parameter, but there was no space in front of the "stage2=..." keyword. Therefore, I got a message that the kickstart file could not be found. Adding a space before stage2 in grub.conf solved the problem.

(The system is upgrading now, so I cannot report the version etc, sorry)

It was preupgrade-1.1.7-1.fc12.noarch.

I tested with preupgrade-1.1.8-1.fc12.1 and this issue is fixed. Once preupgrade-1.1.8-1.fc12.1 hits updates this bug report can be closed.

preupgrade-1.1.8-1.fc12.1 was pushed to stable, closing.