Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1490505

Summary: cloud-init fails when calling `xfs_growfs /dev/mapper/atomicos-root` on Fedora Atomic Host
Product: [Fedora] Fedora Reporter: Dusty Mabe <dustymabe>
Component: cloud-initAssignee: Garrett Holmstrom <gholms>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 27CC: adimania, apevec, dustymabe, esandeen, gholms, kparal, lars, shardy, s
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: AcceptedFreezeException
Fixed In Version: cloud-init-0.7.9-9.fc27 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-09-25 18:05:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1396703    

Description Dusty Mabe 2017-09-11 19:36:25 UTC
Description of problem:

cloud-init fails when calling `xfs_growfs /dev/mapper/atomicos-root`:

```
Sep 11 18:15:50 localhost.localdomain cloud-init[729]: 2017-09-11 18:15:50,299 - util.py[WARNING]: Failed to resize filesystem (cmd=('xfs_growfs', '/dev/mapper/atomicos-root'))
Sep 11 18:15:50 localhost.localdomain cloud-init[729]: 2017-09-11 18:15:50,301 - util.py[WARNING]: Running module resizefs (<module 'cloudinit.config.cc_resizefs' from '/usr/lib/python3.6/site-packages/cloudinit/config/cc_resizefs.py'>) failed
```

Running it manually shows:

```
[root@cloudhost ~]# xfs_info /dev/mapper/atomicos-root 
xfs_info: /dev/mapper/atomicos-root is not a mounted XFS filesystem
```

It looks like this change was desired: https://bugzilla.redhat.com/show_bug.cgi?id=1477192

We need to either get cloud-init updated to handle this or get xfs team to officially support this feature.


Version-Release number of selected component (if applicable):

[root@cloudhost ~]# rpm -q cloud-init xfsprogs
cloud-init-0.7.9-8.fc27.noarch
xfsprogs-4.12.0-4.fc27.x86_64

How reproducible:

Always

Steps to Reproduce:
1. Boot https://kojipkgs.fedoraproject.org/compose/branched/Fedora-27-20170910.n.0/compose/CloudImages/x86_64/images/Fedora-Atomic-27-20170910.n.0.x86_64.qcow2

Comment 1 Eric Sandeen 2017-09-11 19:45:22 UTC
Sigh, yes, xfs_growfs has always been documented as acting on a /mount point/ not a device.

We had reports of people doing "xfs_growfs /mount/point" where they forgot that they had unmounted /mount/point and then accidentally ended up growing the filesystem above it, i.e. the root fs.  And without shrink, they were stuck.  Hence the change.

Rejecting directories that aren't mountpoints but still allowing device paths would be a reasonable option, I guess.  too bad that an undocumented behavior ended up coded into cloud-init.  :(

Comment 2 Dusty Mabe 2017-09-11 19:57:27 UTC
(In reply to Eric Sandeen from comment #1)
> Sigh, yes, xfs_growfs has always been documented as acting on a /mount
> point/ not a device.

That is interesting. I guess I would have thought device would have been supported even before mount point. i.e. usually the progression for resizing filesystems was offline resize support and then online resize support. Hence block device as an argument is not an unreasonable thing to expect to work.

I'm not sure how things progressed in XFS but I guess online grow was always supported since the documentation only says specifying a mount point works. 

> 
> We had reports of people doing "xfs_growfs /mount/point" where they forgot
> that they had unmounted /mount/point and then accidentally ended up growing
> the filesystem above it, i.e. the root fs.  And without shrink, they were
> stuck.  Hence the change.

Yeah I understand that, tricky problem. /me would love shrink-ability on XFS. I definitely miss it from ext4. Combining XFS + lvm-thin is just not as straightforward for a normal user to comprehend. 

> 
> Rejecting directories that aren't mountpoints but still allowing device
> paths would be a reasonable option, I guess.  too bad that an undocumented
> behavior ended up coded into cloud-init.  :(

+1, I think accepting a block device (with a filesystem on top) as an argument is something people would expect to work. Rejecting directories that aren't mountpoints but still allowing device paths would prevent the problem you originally described, but allow people to still pass in block devices.

How do you feel about accepting that as an RFE?

Comment 3 Eric Sandeen 2017-09-11 20:21:23 UTC
xfs_growfs has always been exclusively an online operation, so implying that a device works in general isn't desired; it /must/ be mounted.  There is no offline resize.

If we accept (when we accepted ...) a device, it simply found the mount point from it, and used that under the covers.

What information are you gathering from xfs_info?

Comment 4 Dusty Mabe 2017-09-11 20:26:59 UTC
(In reply to Eric Sandeen from comment #3)
> xfs_growfs has always been exclusively an online operation, so implying that
> a device works in general isn't desired; it /must/ be mounted.  There is no
> offline resize.

good to know

> 
> If we accept (when we accepted ...) a device, it simply found the mount
> point from it, and used that under the covers.
> 
> What information are you gathering from xfs_info?

i'm not sure what cloud-init is doing under the covers, but I think it simply just tries to run `xfs_growfs /dev/mapper/atomicos-root` and it fails. I don't think it uses `xfs_info` at all. I assume the same root cause is what is behind xfs_info (BZ1477192). Is that a valid assumption?

Comment 5 Eric Sandeen 2017-09-11 20:29:40 UTC
Oh, so it really does want to grow. .. this bug refers to both actions, so wasn't sure...

Ok, let me huddle with my xfs dev compatriots and figure out where to go from here...

Comment 6 Dusty Mabe 2017-09-11 20:37:30 UTC
(In reply to Eric Sandeen from comment #5)
> Oh, so it really does want to grow. .. this bug refers to both actions, so
> wasn't sure...

oops - I meant to copy the output from the xfs_growfs command in the original bug description. Either way they both "fail" the same way:

```
[root@cloudhost ~]# xfs_info /dev/mapper/atomicos-root 
xfs_info: /dev/mapper/atomicos-root is not a mounted XFS filesystem
[root@cloudhost ~]# xfs_growfs /dev/mapper/atomicos-root 
xfs_growfs: /dev/mapper/atomicos-root is not a mounted XFS filesystem
```

> 
> Ok, let me huddle with my xfs dev compatriots and figure out where to go
> from here...

Thanks! I'm hoping to solve this problem for Fedora 27 so I appreciate you getting back to me so soon.

Comment 7 Eric Sandeen 2017-09-11 20:58:53 UTC
(I see that yes, cloud-init really does want to grow the filesystem not just get info w/ xfs_info).

And looking further, I think that this is all that's needed for cloud-init to invoke xfs_growfs properly:



diff --git a/cloudinit/config/cc_resizefs.py b/cloudinit/config/cc_resizefs.py
index ceee952..d78e17f 100644
--- a/cloudinit/config/cc_resizefs.py
+++ b/cloudinit/config/cc_resizefs.py
@@ -54,7 +54,7 @@ def _resize_ext(mount_point, devpth):
 
 
 def _resize_xfs(mount_point, devpth):
-    return ('xfs_growfs', devpth)
+    return ('xfs_growfs', mount_point)
 
 
 def _resize_ufs(mount_point, devpth):

Comment 8 Fedora Blocker Bugs Application 2017-09-12 13:16:18 UTC
Proposed as a Freeze Exception for 27-beta by Fedora user dustymabe using the blocker tracking app because:

 This is mostly a cosmetic bug for Atomic Host. For the cloud base image we use ext4 so we aren't affected there. It would be nice to have cloud-init working properly for xfs filesystems; proposing as freeze exception.

Comment 9 Dusty Mabe 2017-09-12 18:38:39 UTC
(In reply to Eric Sandeen from comment #7)
>
>  def _resize_xfs(mount_point, devpth):
> -    return ('xfs_growfs', devpth)
> +    return ('xfs_growfs', mount_point)
>  

seems to work. we'll try to propose to upstream cloud-init

Comment 10 Eric Sandeen 2017-09-12 18:41:11 UTC
feel free to cc: me if you like, and/or simply quote the man page ;)

Comment 11 Dusty Mabe 2017-09-13 19:15:03 UTC
upstream pull request: https://code.launchpad.net/~dustymabe/cloud-init/+git/cloud-init/+merge/330701

Comment 12 Kamil Páral 2017-09-18 17:09:32 UTC
Discussed at blocker review meeting [1]:

AcceptedFreezeException - This seems to be small change which will fix cloud-init for xfs filesystems

[1] https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2017-09-18

Comment 13 Fedora Update System 2017-09-18 22:37:35 UTC
cloud-init-0.7.9-9.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-62a8a2b453

Comment 14 Fedora Update System 2017-09-18 22:37:53 UTC
cloud-init-0.7.9-9.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-5507234819

Comment 15 Fedora Update System 2017-09-18 22:38:25 UTC
cloud-init-0.7.9-9.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-d1613e2c23

Comment 16 Kamil Páral 2017-09-19 13:21:32 UTC
(In reply to Fedora Update System from comment #15)
> cloud-init-0.7.9-9.fc27 has been submitted as an update to Fedora 27.
> https://bodhi.fedoraproject.org/updates/FEDORA-2017-d1613e2c23

I you want to see this pushed to Beta, please make sure it's tested (either here or in bodhi). Thanks.

Comment 17 Dusty Mabe 2017-09-19 21:13:26 UTC
(In reply to Kamil Páral from comment #16)
> (In reply to Fedora Update System from comment #15)
> > cloud-init-0.7.9-9.fc27 has been submitted as an update to Fedora 27.
> > https://bodhi.fedoraproject.org/updates/FEDORA-2017-d1613e2c23
> 
> I you want to see this pushed to Beta, please make sure it's tested (either
> here or in bodhi). Thanks.

I added karma in bodhi (the update is locked right now though). Please test using the following images and also add karma in bodhi:


    https://dustymabe.fedorapeople.org/cloud-init-0.7.9-9-atomic.qcow2
    https://dustymabe.fedorapeople.org/cloud-init-0.7.9-9.qcow2

9272e54c6303112f03be4dfa5dc06bf2  cloud-init-0.7.9-9-atomic.qcow2
3f850a752b59434c4fed6ebbd99bc3d5  cloud-init-0.7.9-9.qcow2

Comment 18 Fedora Update System 2017-09-19 23:28:50 UTC
cloud-init-0.7.9-9.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-d1613e2c23

Comment 19 Fedora Update System 2017-09-20 00:23:13 UTC
cloud-init-0.7.9-9.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-62a8a2b453

Comment 20 Fedora Update System 2017-09-20 01:21:38 UTC
cloud-init-0.7.9-9.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-5507234819

Comment 21 Fedora Update System 2017-09-25 18:05:49 UTC
cloud-init-0.7.9-9.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.