Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1430511 - cloud init doesn't setup ssh keys for access
Summary: cloud init doesn't setup ssh keys for access
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: cloud-init
Version: 26
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Garrett Holmstrom
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedBlocker
Depends On:
Blocks: F26AlphaBlocker
TreeView+ depends on / blocked
 
Reported: 2017-03-08 19:59 UTC by Kevin Fenzi
Modified: 2017-03-20 22:19 UTC (History)
13 users (show)

Fixed In Version: cloud-init-0.7.9-4.fc26
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-03-20 22:19:49 UTC
Type: Bug
Embargoed:

Attachments (Terms of Use)

Description Kevin Fenzi 2017-03-08 19:59:01 UTC 
Using: 

https://dl.fedoraproject.org/pub/fedora/linux/development/26/CloudImages/x86_64/images/Fedora-Cloud-Base-26-20170308.n.0.x86_64.qcow2

in openstack and specifying a ssh keypair on launch, the instance boots fine, but my ssh key doesn't seem to have been setup and I cannot access it via ssh at all. 

The console logs have: 
%G%G%G%G%G%G[    5.828946] cloud-init[324]: Cloud-init v. 0.7.9 running 'init-local' at Wed, 08 Mar 2017 19:52:18 +0000. Up 5.48 seconds.
[[0;32m  OK  [0m] Started Initial cloud-init job (pre-networking).
[[0;32m  OK  [0m] Reached target Cloud-config availability.
[[0;32m  OK  [0m] Reached target System Initialization.
[[0;32m  OK  [0m] Started dnf makecache timer.
[[0;32m  OK  [0m] Started Daily Cleanup of Temporary Directories.
[[0;32m  OK  [0m] Reached target Timers.
[[0;32m  OK  [0m] Listening on D-Bus System Message Bus Socket.
[[0;32m  OK  [0m] Reached target Sockets.
[[0;32m  OK  [0m] Reached target Basic System.
         Starting OpenSSH ed25519 Server Key Generation...
         Starting Login Service...
         Starting NTP client/server...
[[0;32m  OK  [0m] Started D-Bus System Message Bus.
         Starting OpenSSH ecdsa Server Key Generation...
         Starting OpenSSH rsa Server Key Generation...
         Starting Permit User Sessions...
[[0;32m  OK  [0m] Reached target Network (Pre).
         Starting LSB: Bring up/down networking...
[[0;1;39m INFO [0m] Network Service is not active.
[[0;1;33mDEPEND[0m] Dependency failed for Wait for Network to be Configured.
[[0;32m  OK  [0m] Started Permit User Sessions.
[[0;32m  OK  [0m] Started Login Service.
         Starting Hold until boot process finishes up...
[[0;32m  OK  [0m] Started Command Scheduler.
         Starting Terminate Plymouth Boot Screen...
[[0;32m  OK  [0m] Started NTP client/server.
[   10.067425] cloud-init[688]: Cloud-init v. 0.7.9 running 'modules:config' at Wed, 08 Mar 2017 19:52:22 +0000. Up 9.90 seconds.

Fedora 26 (Cloud Edition)
Kernel 4.11.0-0.rc1.git0.1.fc26.x86_64 on an x86_64 (ttyS0)

host-172-25-176-12 login: [   36.187710] cloud-init[697]: Cloud-init v. 0.7.9 running 'modules:final' at Wed, 08 Mar 2017 19:52:23 +0000. Up 10.99 seconds.
[   36.189335] cloud-init[697]: 2017-03-08 19:52:49,166 - util.py[WARNING]: Running module ssh-authkey-fingerprints (<module 'cloudinit.config.cc_ssh_authkey_fingerprints' from '/usr/lib/python3.6/site-packages/cloudinit/config/cc_ssh_authkey_fingerprints.py'>) failed
<14>Mar  8 19:52:49 ec2: 
<14>Mar  8 19:52:49 ec2: #############################################################
<14>Mar  8 19:52:49 ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
<14>Mar  8 19:52:49 ec2: 256 SHA256:Y7P8KOMPsduJ8latwGwp4ZSUhDttD0bl0ue0x7UX7P4 no comment (ECDSA)
<14>Mar  8 19:52:49 ec2: 256 SHA256:5448f014uPjyjWII288XWd6C0X2Y1B0CS9SBuIhaMrY no comment (ED25519)
<14>Mar  8 19:52:49 ec2: 2048 SHA256:UcvTizC6BDuyZLnLZHid9XJMp9U/xI8YVAf4LVnHyJQ no comment (RSA)
<14>Mar  8 19:52:49 ec2: -----END SSH HOST KEY FINGERPRINTS-----
<14>Mar  8 19:52:49 ec2: #############################################################
-----BEGIN SSH HOST KEY KEYS-----
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPf7yZK84hEB9BnDR4TRWURQqyO7iIkpp0PIqk4CfowgD82Fhik37HEPQ2f1B1LfZiXXQqD+1c9+MZYNpDE26aM= 
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMvWq7MIKc6AJTiI/79PoRdf5zKgygxljNC95Zy8/4eB 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCaBwZFRJt1GxUM6yTxe5hS46EwoQkNjBqHBsiwwvIfpf3iLzMlD3lFWMUunaZp2w8NZ0p48hchc0Jwp1+Nw5knkiF7J8dcznxMOXw8r5ap8J7kYFw/4H/JALT2xqGk2G/v7rc0Qj61I3qrhn3NH5jpMTBTOQET5+RbIS6fJwQPC06eG1W3FjqNEJ1/tiztHot7XHDpsSJPZW3QerAFrCIdX1cFN9hLIiXiksmQ2ly+rLDA99EAUcPM/dXaQUf6Tw5DpH0CGnqpR0hn99J+/VFk/N4zOl+GbxHSa8nCKWclJrIrffWm/0djfxQDpZ0zVI7iJLyYaGqFapQZWPb3+O2b 
-----END SSH HOST KEY KEYS-----
[   36.266397] cloud-init[697]: Cloud-init v. 0.7.9 finished at Wed, 08 Mar 2017 19:52:49 +0000. Datasource DataSourceNone.  Up 36.25 seconds
[   36.267733] cloud-init[697]: 2017-03-08 19:52:49,245 - cc_final_message.py[WARNING]: Used fallback datasource

Perhaps the 
[   36.189335] cloud-init[697]: 2017-03-08 19:52:49,166 - util.py[WARNING]: Running module ssh-authkey-fingerprints (<module 'cloudinit.config.cc_ssh_authkey_fingerprints' from '/usr/lib/python3.6/site-packages/cloudinit/config/cc_ssh_authkey_fingerprints.py'>) failed
is related. 

Happy to try and gather more info, but I cannot login to the instance. ;(
Comment 1 Mike Ruckman 2017-03-08 20:04:26 UTC 
Looks like cloud-init is also not setting other things from the meta-data: launching the instance with testcloud doesn't allow you to log in either.
Comment 2 Lars Kellogg-Stedman 2017-03-08 20:08:23 UTC 
A good way to debug this sort of problem is to create a new image that has a password set on the root account.  Assuming you have the image available locally, you can run set the root password to "secret" like this:

  virt-customize -a Fedora-Cloud-Base-26-20170308.n.0.x86_64.qcow2 --root-password password:secret


...and then upload the image to glance and boot from it.

---

I tested this out locally and I can confirm the problem.  There is a dependency cycle involving cloud-init.service that is causing systemd to delete the cloud-init.service job:

# journalctl -u sysinit.target -b |cat
-- Logs begin at Wed 2017-03-08 20:04:27 UTC, end at Wed 2017-03-08 20:07:52 UTC. --
Mar 08 20:06:18 localhost systemd[1]: Reached target System Initialization.
Mar 08 20:06:19 localhost systemd[1]: Stopped target System Initialization.
Mar 08 20:06:19 localhost.localdomain systemd[1]: sysinit.target: Found ordering cycle on sysinit.target/start
Mar 08 20:06:19 localhost.localdomain systemd[1]: sysinit.target: Found dependency on cloud-init.service/start
Mar 08 20:06:19 localhost.localdomain systemd[1]: sysinit.target: Found dependency on basic.target/start
Mar 08 20:06:19 localhost.localdomain systemd[1]: sysinit.target: Found dependency on sockets.target/start
Mar 08 20:06:19 localhost.localdomain systemd[1]: sysinit.target: Found dependency on dbus.socket/start
Mar 08 20:06:19 localhost.localdomain systemd[1]: sysinit.target: Found dependency on sysinit.target/start
Mar 08 20:06:19 localhost.localdomain systemd[1]: sysinit.target: Breaking ordering cycle by deleting job cloud-init.service/start
Mar 08 20:06:21 localhost.localdomain systemd[1]: Reached target System Initialization.
Comment 3 Lars Kellogg-Stedman 2017-03-08 20:15:30 UTC 
I think the best way of resolving this is to remove the cloud-init dependency on Before=sysinit.target. There are simply too many things that want to come *after* that point.

You can see the patches I've applied against 0.7.9 for the RedHat package here:

  https://github.com/larsks/cloud-init/compare/0.7.9...0.7.9-patches

Particularly relevant to this bz:

  https://github.com/larsks/cloud-init/commit/3f8d6dbbbc9ab4679a4820d7cc60265fa67807cd
Comment 4 Mike Ruckman 2017-03-13 19:51:45 UTC 
Testing against the 20170313 builds, I don't see this issue.
Comment 5 Dusty Mabe 2017-03-13 19:56:40 UTC 
(In reply to Mike Ruckman from comment #4)
> Testing against the 20170313 builds, I don't see this issue.

This is probably because systemd can pick/choose one of two services to remove in order to clear the dependency loop. This will most likely succeed sometimes and fail others. We need to get this fixed and are hoping to have a new rpm this week to test.
Comment 6 Mike Ruckman 2017-03-13 19:57:52 UTC 
I used the wrong image for this. False alarm, sorry for the noise. :(
Comment 7 Lars Kellogg-Stedman 2017-03-14 15:00:33 UTC 
There is a scratch build @ https://koji.fedoraproject.org/koji/taskinfo?taskID=18380550 that should resolve this problem.  Please give it a try and let me know if it works for you.
Comment 8 Kevin Fenzi 2017-03-14 18:57:05 UTC 
I'm not sure I have a handy way to rebuild a cloud image with a scratch build. 

Will see what I can do, but if anyone else wants to test first, go for it.
Comment 9 Dusty Mabe 2017-03-14 19:01:59 UTC 
(In reply to Kevin Fenzi from comment #8)
> I'm not sure I have a handy way to rebuild a cloud image with a scratch
> build. 
> 
> Will see what I can do, but if anyone else wants to test first, go for it.

hey kevin. I think the goal is to get gholms to take the srpm from that scratch build and pull the patches out of it and apply them to dist-git and build a new cloud-init for f26.
Comment 10 Fedora Update System 2017-03-14 23:35:16 UTC 
cloud-init-0.7.9-4.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-83671c0fa0
Comment 11 Fedora Update System 2017-03-15 04:24:50 UTC 
cloud-init-0.7.9-4.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-83671c0fa0
Comment 12 Kevin Fenzi 2017-03-16 01:49:10 UTC 
ok. I tried todays rawhde cloud base: 

http://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/CloudImages/x86_64/images/Fedora-Cloud-Base-Rawhide-20170315.n.0.x86_64.qcow2

and it does work, but it takes quite a while to come up... 

..
[[0;32m  OK  [0m] Reached target Network (Pre).
         Starting LSB: Bring up/down networking...
[[0;1;39m INFO [0m] Network Service is not active.
[[0;1;33mDEPEND[0m] Dependency failed for Wait for Network to be Configured.
...
 Starting Initial cloud-init job (metadata service crawler)...
[  230.547893] cloud-init[691]: Cloud-init v. 0.7.9 running 'init' at Thu, 16 Mar 2017 01:42:06 +0000. Up 11.79 seconds
...
-----END SSH HOST KEY KEYS-----
[  232.955374] cloud-init[796]: Cloud-init v. 0.7.9 running 'modules:final' at Thu, 16 Mar 2017 01:45:47 +0000. Up 232.72 seconds.
[  232.957588] cloud-init[796]: Cloud-init v. 0.7.9 finished at Thu, 16 Mar 2017 01:45:47 +0000. Datasource DataSourceOpenStack [net,ver=2].  Up 232.94 seconds
Comment 13 Adam Williamson 2017-03-16 02:09:06 UTC 
This is in fact probably an Alpha blocker, per criterion "Release-blocking cloud images must allow login with the user authentication configuration requested during instance creation" - https://fedoraproject.org/wiki/Fedora_26_Alpha_Release_Criteria#Expected_image_boot_behavior . https://fedoraproject.org/wiki/Releases/26/ReleaseBlocking still lists two Cloud images as release blocking for the main release cycle, regardless of the existence of two-week Atomic.
Comment 14 Kevin Fenzi 2017-03-16 02:52:01 UTC 
Yeah, sadly I think that page is not up to date, but we can only go on what we have... so +1 blocker.
Comment 15 Mike Ruckman 2017-03-16 02:56:23 UTC 
Yeah, this one fits the criteria. +1
Comment 16 Adam Williamson 2017-03-16 03:05:39 UTC 
That's three votes, marking as an accepted blocker.
Comment 17 Fedora Update System 2017-03-20 22:19:49 UTC 
cloud-init-0.7.9-4.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.
Note You need to log in before you can comment on or make changes to this bug.