Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1569321
Summary: | cloud-init-17.1-4.fc28 sets BOOTPROTO=none when using EC2 private network only | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Joe Doss <joe> | ||||||
Component: | cloud-init | Assignee: | Garrett Holmstrom <gholms> | ||||||
Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||
Severity: | unspecified | Docs Contact: | |||||||
Priority: | unspecified | ||||||||
Version: | 28 | CC: | adimania, apevec, awilliam, dustymabe, fzatlouk, gholms, kevin, lars, miabbott, ngompa13, pwhalen, robatino, sayan.chowdhury2012, sayanchowdhury, shardy, s | ||||||
Target Milestone: | --- | ||||||||
Target Release: | --- | ||||||||
Hardware: | Unspecified | ||||||||
OS: | Unspecified | ||||||||
Whiteboard: | AcceptedFreezeException | ||||||||
Fixed In Version: | cloud-init-17.1-5.fc28 | Doc Type: | If docs needed, set a value | ||||||
Doc Text: | Story Points: | --- | |||||||
Clone Of: | Environment: | ||||||||
Last Closed: | 2018-04-25 00:02:32 UTC | Type: | Bug | ||||||
Regression: | --- | Mount Type: | --- | ||||||
Documentation: | --- | CRM: | |||||||
Verified Versions: | Category: | --- | |||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||
Embargoed: | |||||||||
Bug Depends On: | |||||||||
Bug Blocks: | 1469206, 1469207 | ||||||||
Attachments: |
|
Description
Joe Doss
2018-04-19 03:58:03 UTC
Proposed as a Blocker for 28-final by Fedora user dustymabe using the blocker tracking app because: "The cloud-init package must be functional for release blocking cloud images." I don't know if this is really a blocker (per release criteria above), but we have found a case in ec2 where cloud-init doesn't work for instances that are requested to not have a public IP (i.e. access to them is through other instances in the same VPC). Proposing as blocker for now and then we can just bump to FE if it doesn't meet the requirements. I'm at least +1 FE to this, not entirely sure if it merits blocker status, don't have enough experience of how big a deal this is. To add some end-user perspective. We have ~200 VMs running Fedora Cloud and six of them use the public IP features in EC2. Since we rely heavily on running VMs with only private IP addresses (we access them via SSH jump box) this makes this bug huge blocker on using Fedora 28 Atomic / Cloud out of the box. I understand that it might not fit all the release criteria for a blocker, but I wanted add some color on why this could be a blocker for some end users on consuming Fedora 28 on release. The good news is that I built an AMI with a downgraded cloud-init and it works just fine. [jdoss@sts71 ~]$ ssh fedora.11.156 Warning: Permanently added '10.0.11.156' (ECDSA) to the list of known hosts. [fedora@ip-10-0-11-156 ~]$ cat /etc/redhat-release Fedora release 28 (Twenty Eight) [fedora@ip-10-0-11-156 ~]$ rpm -qa |grep cloud-init cloud-init-0.7.9-9.fc27.noarch Maybe just reverting cloud-init back down to 0.7.9-9 is quick path to getting this issue fixed until cloud-init-17.1-4 gets sorted out. Created attachment 1424309 [details] cloud-init.log attached cloud-init log from https://pagure.io/atomic-wg/issue/456 Created attachment 1424310 [details] journal.log attached system journal from https://pagure.io/atomic-wg/issue/456 Inspecting the obj.pkl file created by cloud-init on a failing instance, we see: >>> import cloudinit >>> import pickle >>> d=pickle.load(open('obj.pkl','rb')) >>> >>> d.network_config {'version': 1, 'config': [{'type': 'physical', 'name': 'eth0', 'subnets': [], 'mac_address': '12:66:92:11:cc:64'}]} While on an instance with a public ip address, we see: >>> import cloudinit >>> import pickle >>> d=pickle.load(open('obj.pkl','rb')) >>> d.network_config {'version': 1, 'config': [{'type': 'physical', 'name': 'eth0', 'subnets': [{'type': 'dhcp4'}], 'mac_address': '12:ae:7e:61:ab:e8'}]} Note the difference in the contents of the "subnets" key, which is empty on the failing instance. This appears to be a bug in cloud-init that was fixed upstream in https://git.launchpad.net/cloud-init/commit/?id=eb292c18c3d83b9f7e5d1fd81b0e8aefaab0cc2d. Scratch build @ https://koji.fedoraproject.org/koji/taskinfo?taskID=26485084 that should resolve the problem. Someone let me know if it works? This worked for my test. I created ami-da8a21a5. Please others use this to verify things work! Lars, can you create a PR to cloud-init? and we'll get it merged and we'll create an official build and get a bodhi update for it. I've opened https://src.fedoraproject.org/rpms/cloud-init/pull-request/2 with the fix. This worked with my tests as well: [jdoss@sts71 ~]$ ssh fedora.11.104 Warning: Permanently added '10.0.11.104' (ECDSA) to the list of known hosts. [fedora@ip-10-0-11-104 ~]$ cat /etc/redhat-release Fedora release 28 (Twenty Eight) [fedora@ip-10-0-11-104 ~]$ rpm -qa |grep cloud-init cloud-init-17.1-5.fc29.noarch I am able to connect to the instance over the private network without any public IP added to the server. cloud-init-17.1-5.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-768f665d9d Discussed during the 2018-04-23 blocker review meeting: [1] The decision to classify this bug as an AcceptedFreezeException and punt (delay decision) for Blocker was made: "this is a judgment call as it depends how common ec2 instances with only private IPs are. We agree this is at least serious enough to be an FE; we don't yet have enough data to be totally sure if it's serious enough to be a blocker. If the fix presents problems, we will revisit whether to make this a blocker." [1] https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2018-04-23/f28-blocker-review.2018-04-23-16.00.log.txt cloud-init-17.1-5.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-768f665d9d I don't have a handy way to test this in openstack without a .qcow2, but I did just test today's rawhide image that has cloud-init-17.1-5.fc29.noarch with these same changes in it. It works as expected no problems with network. So, at least it does not appear to cause any regressions with the openstack case. This issue was afflicting me too, I'm looking forward to this being fixed. :) I created a raw image [1] and corresponding ami-b48236cb (us-east-1) for people to use to test the fix for this bug. [1] https://dustymabe.fedorapeople.org/cloud-init-bz1569321-f28.raw (In reply to Dusty Mabe from comment #17) > I created a raw image [1] and corresponding ami-b48236cb (us-east-1) for > people to use to test the fix for this bug. > > [1] https://dustymabe.fedorapeople.org/cloud-init-bz1569321-f28.raw scratch that.. use this raw.xz image link instead: https://dustymabe.fedorapeople.org/cloud-init-bz1569321-f28.raw.xz Tested on a local private network, no issues using https://dustymabe.fedorapeople.org/cloud-init-bz1569321-f28.raw.xz Paul: can you give the update karma too, then? Thanks! I tested Dusty's new AMI in us-east-1 and in us-west-2 and it works as intended. [jdoss@sts71 ~]$ ssh fedora.11.82 Warning: Permanently added '10.0.11.82' (ECDSA) to the list of known hosts. [fedora@ip-10-0-11-82 ~]$ cat /etc/redhat-release Fedora release 28 (Twenty Eight) [fedora@ip-10-0-11-82 ~]$ cloud-init-17.1-5.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report. |