Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1353140 - livemedia-creator creates live cds which have almost no space on root
Summary: livemedia-creator creates live cds which have almost no space on root
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: lorax
Version: 24
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Brian Lane
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-06 08:40 UTC by bollocks
Modified: 2016-07-18 18:24 UTC (History)
4 users (show)

Fixed In Version: lorax-24.20-1 lorax-24.20-1.fc24
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-07-18 18:24:00 UTC
Type: ---
Embargoed:

Attachments (Terms of Use)

Description bollocks 2016-07-06 08:40:35 UTC 
User-Agent:       Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:47.0) Gecko/20100101 Firefox/47.0
Build Identifier: 

Using live media creator from lorax 24.19-1.fc24 I created a live cd based on kickstarts from https://pagure.io/fedora-kickstarts. It created a live .iso file that runs but has about 50MB of space on /.  

Reproducible: Always

Steps to Reproduce:
1.Use ksflatten to make a kickstart file from the fedora-live-lxde.ks file in fedora-kickstarts.
2. grab a copy of http://dl.fedoraproject.org/pub/fedora/linux/releases/24/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-24-1.2.iso and name it netinst24.iso
3.Run livemedia-creator --make-iso --iso netinst24.iso --ks flattenedlxde.ks --logfile logfile.txt --tmp 'directorytouseastmp' --title 'Fedora 24' --project 'F24 LXDE Spin' --releasever 1.112 --ram 640 --vcpus 4
4.Retrieve boot.iso from the directory you used with the --tmp flag.  The output at the end of livemedia-creator will tell you exactly where it is in that directory.
5. Run the image on baremetal or with qemu-kvm -m 2048 -vga qxl -cdrom boot.iso
6. Run df -h to see how full / is or do anything that requires more than a few dozen MB.


Actual Results:  
After running dnf makecache dnf reported an error.  Running df -h returned the following results

Filesystem           Size  Used Avail Use% Mounted on
devtmpfs             981M     0  981M   0% /dev
tmpfs               1001M  144K 1001M   1% /dev/shm
tmpfs               1001M  816K 1000M   1% /run
tmpfs               1001M     0 1001M   0% /sys/fs/cgroup
/dev/sr0             947M  947M     0 100% /run/initramfs/live
/dev/mapper/live-rw  2.8G  2.8G  992K 100% /
tmpfs               1001M   16K 1001M   1% /tmp
vartmp              1001M     0 1001M   0% /var/tmp
tmpfs                201M   12K  201M   1% /run/user/1000


Expected Results:  
Usually when I make a live cd like this I have way more room than I need to run dnf makecache.  There's usually enough space to install a few dozen packages or updates.  

I ran in to this problem when trying to update an i3 fedora remix that I base off of the LXDE spin so I have two closely related cases where this occurs.
Comment 1 bollocks 2016-07-06 10:53:47 UTC 
I ran it again and saw a red error message fly across the screen on bootup.  I checked the journal and I suspect this might have something to do with it.  It looks like the squashfs image is getting mounted but when it goes to make the overlay and mount it something isn't right.

journalctl | grep systemd-remount 

Jul 06 10:37:46 localhost.localdomain systemd-remount-fs[745]: mount: can't find UUID=e7214dd7-d6c6-4a0e-918d-3b401c714415
Jul 06 10:37:46 localhost.localdomain systemd-remount-fs[745]: /usr/bin/mount for / exited with exit status 1.
Jul 06 10:37:46 localhost.localdomain systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE
Jul 06 10:37:46 localhost.localdomain systemd[1]: systemd-remount-fs.service: Unit entered failed state.
Jul 06 10:37:46 localhost.localdomain audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-remount-fs comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Jul 06 10:37:46 localhost.localdomain systemd[1]: systemd-remount-fs.service: Failed with result 'exit-code'.

This is an image created back when Fedora 24 RC 1.2 first came out:
https://bollocks.fedorapeople.org/i3spin/F24-x86_64-I3-1.111.iso

I get the same error with this image whether in qemu-kvm or from a usb stick on baremetal.  

Here's the result of ksflatten for fedora-live-lxde.ks from the fedora-kickstarts pagure git: https://bollocks.fedorapeople.org/gitlxdespin.ks
Comment 2 bollocks 2016-07-06 11:03:08 UTC 
By the way that journal excerpt is from the remix I linked.  I decided to check with the lxde image made with livemedia-creator.  There's a similar error

journalctl | grep systemd-remount

Jul 06 10:55:22 localhost.localdomain systemd-remount-fs[754]: mount: can't find UUID=26b5df21-7ba2-45d4-8bf9-e190b35893e6
Jul 06 10:55:22 localhost.localdomain systemd-remount-fs[754]: /usr/bin/mount for / exited with exit status 1.
Jul 06 10:55:22 localhost.localdomain audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-remount-fs comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Jul 06 10:55:22 localhost.localdomain systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE
Jul 06 10:55:22 localhost.localdomain systemd[1]: systemd-remount-fs.service: Unit entered failed state.
Jul 06 10:55:22 localhost.localdomain systemd[1]: systemd-remount-fs.service: Failed with result 'exit-code'.

I tried the official LXDE spin release and saw no error.  df -h reports 2.3GB available on /.  The excerpt from the journal:

journalctl | grep systemd-remount

Jul 06 10:59:09 localhost audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-remount-fs comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Comment 3 Brian Lane 2016-07-07 18:30:06 UTC 
Does your kickstart remove /etc/fstab in %post? If it doesn't then dracut/systemd will get confused looking for a UUID that doesn't exist.

Also, the size of the root filesystem is controlled by the 'part / --size XXX' command in the kickstart, increasing that value should give you more room.
Comment 4 bollocks 2016-07-07 19:16:49 UTC 
There is a weird UUID entry in fstab in the livecds.  I'll delete fstab right before the 

cat >> /etc/fstab << EOF
vartmp   /var/tmp    tmpfs   defaults   0  0
EOF

section so that part of the kickstart is still allowed to do what they intended.


section in the LXDE kickstart and run it again.  The part / --size is set to 5GB. However I thought when making a LiveCD that only set the size of the file system that it creates to install the packages on.  It seems to get resized and then sent to a squashfs first.  Either way the squashfs is 900MB and the unsquashed version is 2.8GB so 5GB shouldn't be limiting it.  I'll report back after it runs.
Comment 5 bollocks 2016-07-07 20:57:07 UTC 
I created run livemedia-creator with the above edit.  The edited kickstart can be found here:
https://bollocks.fedorapeople.org/gitlxdespinv2.ks

The generated iso only has 90MB of space on /.  I can't run dnf makecache without running out of space.  It does get rid of the systemd-remount error.  

I'm going to remove fstab completely by getting rid of the vartmp section in %post just to see what happens. The comment in the kickstart says it was added to fix a bug back in F21.
Comment 6 Brian Lane 2016-07-07 22:24:29 UTC 
Ok, I see what's happening. I fixed this on master with commit 4699c881090b55de12bfe6b8548b3bfe30207ba8, but haven't backported it to f24-branch yet. You're right, in 24.19 it shrinks the ext4 fs down to the minumum after doing the installation.
Comment 7 Brian Lane 2016-07-08 00:34:41 UTC 
https://github.com/rhinstaller/lorax/pull/137
Comment 8 bollocks 2016-07-08 02:12:09 UTC 
I ported the stuff from the master branch on git hub in to lorax and rebuilt the rpms.  It looks like I did it right based on that link.  I started it about 90 minutes ago.  For some reason mksquashfs is taking a long time.  It has been running for 55 minutes where it usually takes 20 minutes.  It got called with 

-comp xz -Xbcj x86

I remember xz compression being used but I don't remember a filter.  I'm not sure that it wasn't there before.  top is only reporting 130% cpu use where normally it report about 360%.  It is making progress though.  I can see the output file growing.
Comment 9 bollocks 2016-07-08 03:21:59 UTC 
It seems to be working.  I was able to run dnf makecache and install firefox which is another 140MB or so.  

I missed the divide by 1024 in the master branch.  So I ended up creating a 5TB file system. lol  That may be what happened with the long time.  I'll apply the whole fix and try again.  I'm pretty sure it will work.
Comment 10 bollocks 2016-07-08 06:04:35 UTC 
The fix works.  Thanks. I'm not sure if I should close the bug or wait until the fix makes it to the repo.  I put the fix up on copr:

https://copr.fedorainfracloud.org/coprs/bollocks/lorax/
Comment 11 Brian Lane 2016-07-08 14:56:44 UTC 
There will be a new build today.
Comment 12 Fedora Update System 2016-07-09 00:27:23 UTC 
lorax-24.20-1.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-6e292f9d34
Comment 13 bollocks 2016-07-09 20:53:09 UTC 
I grabbed it from koji and it works great.  Thanks.
Comment 14 Fedora Update System 2016-07-10 15:57:39 UTC 
lorax-24.20-1.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-6e292f9d34
Comment 15 Fedora Update System 2016-07-18 18:23:58 UTC 
lorax-24.20-1.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.
Note You need to log in before you can comment on or make changes to this bug.