Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1447777 - Since curl-7.53.1-7.fc26 , curl TLS transactions in initramfs environment are broken
Summary: Since curl-7.53.1-7.fc26 , curl TLS transactions in initramfs environment are...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: dracut
Version: 26
Hardware: All
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: dracut-maint-list
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedBlocker
: 1447276 (view as bug list)
Depends On:
Blocks: PPCTracker F26BetaBlocker F26PPCBeta
TreeView+ depends on / blocked
 
Reported: 2017-05-03 19:28 UTC by Adam Williamson
Modified: 2017-05-17 11:55 UTC (History)
15 users (show)

Fixed In Version: dracut-044-181.fc26
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-05-17 11:55:17 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Adam Williamson 2017-05-03 19:28:13 UTC
curl-7.53.1-7 changed the default curl source of trust from /etc/pki/tls/certs/ca-bundle.crt to (libdir)/libnssckbi.so:

http://pkgs.fedoraproject.org/cgit/rpms/curl.git/commit/?id=9549974a4cbd8216ce694e85ce69aa128ff939dc

since that change, no TLS transactions using curl have worked in the initramfs environment, because that file is missing from it.

The "obvious" fix is this:

[adamw@adam dracut (curl-nssckbi %)]$ git diff
diff --git a/modules.d/45url-lib/module-setup.sh b/modules.d/45url-lib/module-setup.sh
index 1ece400f..0fcf7f91 100755
--- a/modules.d/45url-lib/module-setup.sh
+++ b/modules.d/45url-lib/module-setup.sh
@@ -20,6 +20,7 @@ install() {
     inst_multiple -o ctorrent
     inst_multiple curl
     # also install libs for curl https
+    inst_libdir_file "libnssckbi.so*"
     inst_libdir_file "libnsspem.so*"
     inst_libdir_file "libnsssysinit.so*"
     inst_libdir_file "libsoftokn3.so*"

but I'm not sure if that will be tripped up by the file being a symlink. I'm going to run some tests now and will file a dracut PR if I can verify that i have a working fix.

Nominating as Fedora 26 Beta release blocker, because this prevents the use of any kickstart or updates image on an HTTPS server:

"The scripted installation mechanism must provide a working function for creating local user accounts, including the ability to specify a hashed password, and for specifying a hashed password for the root account."

"The installer must be able to download and use an installer update image from an HTTP server."

https://fedoraproject.org/wiki/Fedora_26_Alpha_Release_Criteria#Scripted_user_creation

Comment 1 Adam Williamson 2017-05-03 19:29:43 UTC
This is the bug that's breaking the openQA install_kickstart_firewall_configured and install_kickstart_firewall_disabled tests - they are the ones that use an HTTPS-hosted kickstart - and install_scsi_updates_img - that one uses an HTTPS-hosted updates image.

Comment 2 Adam Williamson 2017-05-03 20:15:23 UTC
Good news: inst_libdir_file manages to resolve the symlinks and install the file.

Bad news: the nssckbi route still needs a certificate trust source of some kind, and we're not including it, whatever it is (I think it's /etc/pki/nssdb/cert9.db and /etc/pki/nssdb/key4.db ).

This vaguely terrifying snippet is, apparently, how dracut tries to include the appropriate cert bundle for curl:

    for _dir in $libdirs; do
	[[ -d $_dir ]] || continue
        for _lib in $_dir/libcurl.so.*; do
	    [[ -e $_lib ]] || continue
            _crt=$(grep -F --binary-files=text -z .crt $_lib)
            [[ $_crt ]] || continue
            [[ $_crt == /*/* ]] || continue
            if ! inst "$_crt"; then
                dwarn "Couldn't install '$_crt' SSL CA cert bundle; HTTPS might not work."
                continue
            fi
            _found=1
        done
    done
    [[ $_found ]] || dwarn "Couldn't find SSL CA cert bundle; HTTPS won't work."

from https://git.kernel.org/pub/scm/boot/dracut/dracut.git/tree/modules.d/45url-lib/module-setup.sh#n32 - that's just not doing the right thing for this (new?) codepath in curl at all, I don't think.

Comment 3 Adam Williamson 2017-05-03 22:56:44 UTC
https://github.com/dracutdevs/dracut/pull/226 is my proposal to fix this. It does work in my testing, but there may be improvements that could be made to it.

Comment 4 Kamil Dudka 2017-05-04 07:19:58 UTC
(In reply to Adam Williamson from comment #0)
> but I'm not sure if that will be tripped up by the file being a symlink.

Symlink to what?

> Nominating as Fedora 26 Beta release blocker, because this prevents the use
> of any kickstart or updates image on an HTTPS server:

Sounds reasonable.

Comment 5 Menanteau Guy 2017-05-04 08:31:17 UTC
*** Bug 1447276 has been marked as a duplicate of this bug. ***

Comment 6 Adam Williamson 2017-05-04 15:37:15 UTC
kdudka: libnssckbi.so is handled by alternatives (for some reason) so /usr/lib64/libnssckbi.so is a symlink to /etc/alternatives/libnssckbi.so.x86_64 , which is a symlink to /usr/lib64/pkcs11/p11-kit-trust.so . But just using dracut's 'inst_libdir_file' function works okay, it installs the actual file into the initramfs.

Comment 7 Adam Williamson 2017-05-04 15:38:49 UTC
CCing anaconda folks for information - if you notice / get reports of kickstart installs failing in recent F26, this is likely why.

Comment 8 Kamil Dudka 2017-05-04 15:57:28 UTC
OK, I did not realize it was a symlink.  Then it could probably be switched to the legacy implementation of libnssckbi.so.  Nevertheless, if you made it work with p11-kit-trust.so, it is even better because it provides a more reliable source of trust.

Comment 9 Dusty Mabe 2017-05-08 16:30:18 UTC
I'm seeing this when trying to use the mirrors as a repo for the anaconda install. Sometimes I get an https mirror (in which case I see a failure) and sometimes I get an http mirror (in which case it succeeds). Even worse is the failures happen at different times during the install because i could get one mirror for downloading stage2 and a different one when anaconda is doing software selection and thus if one of those is https things will fail.

Comment 10 Dusty Mabe 2017-05-08 16:36:45 UTC
actually, this should theoretically only affect initramfs so failures later in the install shouldn't be related to this bug

Comment 11 Adam Williamson 2017-05-08 17:06:01 UTC
Well, it's possible the same issue exists in the installer environment I guess, as that gets stripped by lorax in some ways. But I haven't checked on that, and it would theoretically be a separate bug if it was the case (as it won't be fixed by fixing dracut).

Comment 12 Kamil Dudka 2017-05-09 06:50:19 UTC
Where can I download an installation image to debug this?

Comment 13 Adam Williamson 2017-05-09 07:28:05 UTC
https://www.happyassassin.net/nightlies.html

Comment 14 Kamil Dudka 2017-05-09 08:25:41 UTC
I was not able to reproduce the problem using the following image:
https://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170508.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-26-20170508.n.0.iso

After it booted, I switched to terminal using Ctrl + Alt + F2 and typed the following command:
# curl -svo/dev/null https://google.com

It connected over TLS successfully.  The verbose output contained:
* loaded libnssckbi.so

Please provide details on how to trigger the bug mentioned in comment #9.

Comment 15 Dusty Mabe 2017-05-09 13:14:35 UTC
A command like this should do it:

```
virt-install --name builder --ram  1500 --vcpus 2 --disk path=.//builder.img,size=12 --accelerate --location https://download.fedoraproject.org/pub/fedora/linux/development/26/Everything/x86_64/os --graphics none --force --noreboot --network network=default --extra-args=console=ttyS0 ks=https://kojipkgs.fedoraproject.org//work/tasks/748/19420748/koji-f26-build-19420748-base.ks inst.sshd
```

which ends up with a VM that has these errors:

```
[   10.326047] dracut-initqueue[632]: Warning: can't find installer mainimage path in .treeinfo
[   10.342555] dracut-initqueue[632]:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[   10.345015] dracut-initqueue[632]:                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[   10.367653] dracut-initqueue[632]: curl: (77) Problem with the SSL CA cert (path? access rights?)
[   10.374227] dracut-initqueue[632]: Warning: Downloading 'https://download.fedoraproject.org/pub/fedora/linux/development/26/Everything/x86_64/os/images/install.img' failed!
[   10.387305] dracut-initqueue[632]:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[   10.389286] dracut-initqueue[632]:                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[   10.473252] dracut-initqueue[632]: curl: (77) Problem with the SSL CA cert (path? access rights?)
[   10.476304] dracut-initqueue[632]: Warning: Downloading 'https://download.fedoraproject.org/pub/fedora/linux/development/26/Everything/x86_64/os/LiveOS/squashfs.img' failed!
[   10.478516] dracut-initqueue[632]: Warning: Could not retrieve stage2 image from https://download.fedoraproject.org/pub/fedora/linux/development/26/Everything/x86_64/os using ens2
[   11.022875] dracut-initqueue[632]:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[   11.024902] dracut-initqueue[632]:                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[   11.126322] dracut-initqueue[632]: curl: (77) Problem with the SSL CA cert (path? access rights?)
[   11.127208] dracut-initqueue[632]: Warning: failed to fetch kickstart from https://kojipkgs.fedoraproject.org//work/tasks/748/19420748/koji-f26-build-19420748-base.ks
```

Comment 16 Dusty Mabe 2017-05-09 13:16:51 UTC
note that you did not see an error because you were already into anaconda installer. I'm seeing the error before we get to anaconda, in dracut. 


I had some issues with quoting in that previous output. Use this instead:

```
virt-install --name builder --ram  1500 --vcpus 2 --disk path=.//builder.img,size=12 --accelerate --location https://download.fedoraproject.org/pub/fedora/linux/development/26/Everything/x86_64/os --graphics none --force --noreboot --network network=default --extra-args='console=ttyS0 ks=https://kojipkgs.fedoraproject.org//work/tasks/748/19420748/koji-f26-build-19420748-base.ks inst.sshd'
```

Comment 17 Adam Williamson 2017-05-09 15:19:38 UTC
Kamil: that's not the initramfs environment. That's the installer environment.

The easiest way to observe the problem is to just try and use a kickstart or updates image hosted on HTTPS, e.g. using this kernel parameter:

inst.ks=https://fedorapeople.org/groups/qa/kickstarts/firewall-configured-net.ks

it should result in a curl error, anaconda will fail to start, and eventually the system should dump you at a bash prompt in the initramfs environment, from where you can poke around and confirm the problem.

Comment 18 Alessio 2017-05-09 17:13:31 UTC
Using virt-manager, as described here: https://fedoraproject.org/wiki/QA:Testcase_Boot_Methods_Pxeboot

Using vmlinuz and initrd.img from https://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170504.n.1/compose/Workstation/x86_64/os/images/pxeboot/

And using 
console=ttyS0 inst.repo=https://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170504.n.1/compose/Workstation/x86_64/os
as kernel args

I hit the same issue

[    5.463750] 8139cp 0000:00:03.0 ens3: renamed from eth0
[    5.577739] 8139cp 0000:00:03.0 ens3: link up, 100Mbps, full-duplex, lpa 0x05E1
[    8.625156] dracut-initqueue[658]: Warning: can't find installer mainimage path in .treeinfo
[    8.636791] dracut-initqueue[658]:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[    8.637492] dracut-initqueue[658]:                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[    8.842588] dracut-initqueue[658]: curl: (77) Problem with the SSL CA cert (path? access rights?)
[    8.845857] dracut-initqueue[658]: Warning: Downloading 'https://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170504.n.1/compose/Workstation/x86_64/os/images/install.img' failed!
[    8.860042] dracut-initqueue[658]:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[    8.863912] dracut-initqueue[658]:                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[    9.129313] dracut-initqueue[658]: curl: (77) Problem with the SSL CA cert (path? access rights?)
[    9.133672] dracut-initqueue[658]: Warning: Downloading 'https://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170504.n.1/compose/Workstation/x86_64/os/LiveOS/squashfs.img' failed!
[    9.135731] dracut-initqueue[658]: Warning: Could not retrieve stage2 image from https://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170504.n.1/compose/Workstation/x86_64/os using ens3
[   94.732084] random: crng init done
[  132.395467] dracut-initqueue[658]: Warning: dracut-initqueue timeout - starting timeout scripts
[  132.914615] dracut-initqueue[658]: Warning: dracut-initqueue timeout - starting timeout scripts
[  133.440413] dracut-initqueue[658]: Warning: dracut-initqueue timeout - starting timeout scripts
[  133.949967] dracut-initqueue[658]: Warning: dracut-initqueue timeout - starting timeout scripts
...
[  193.311689] dracut-initqueue[658]: Warning: Could not boot.
[  193.423458] dracut-initqueue[658]: Warning: /dev/root does not exist
         Starting Setup Virtual Console...
[  OK  ] Started Setup Virtual Console.
...
Warning: /dev/root does not exist

Generating "/run/initramfs/rdsosreport.txt"


Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.


dracut:/#

Comment 19 Adam Williamson 2017-05-09 19:05:35 UTC
Yes, that would be expected. This bug will affect any codepath which involves downloading via HTTPS from the initramfs environment (before pivot to the installer environment), which obviously includes retrieving the installer environment itself via HTTPS.

Comment 20 Kamil Dudka 2017-05-10 07:24:12 UTC
OK.  I was confused by comment #11 where a (theoretical?) bug in the installer environment was mentioned.  Do I understand it correctly that only dracut is affected by this bug and the bug is going to be fixed by the patch mentioned in comment #3?

Anyway, thank all of you for providing the steps to reproduce!

I am able to debug it locally now:

dracut:/# curl -svo/dev/null https://google.com
* Rebuilt URL to: https://google.com/
*   Trying 172.217.22.14...
* TCP_NODELAY set
* Connected to google.com (172.217.22.14) port 443 (#0)
* Initializing NSS with certpath: none
*   CAfile: none
  CApath: none
* failed to load libnssckbi.so
* Closing connection 0

dracut:/# stat /usr/lib64/libnssckbi.so
stat: cannot stat '/usr/lib64/libnssckbi.so': No such file or directory

dracut:/# stat /etc/pki/tls/certs/ca-bundle.crt
stat: cannot stat '/etc/pki/tls/certs/ca-bundle.crt': No such file or directory

Comment 21 Adam Williamson 2017-05-10 07:31:36 UTC
Yeah, in #c11 we were discussing that the same issue *may* exist in the installer environment, but it doesn't look like that's the case.

Note that /etc/pki/tls/certs/ca-bundle.crt is not, in fact, the important bundle - that's the bundle curl *previously* used, but with libnssckbi , the important bundle file is /usr/share/pki/ca-trust-source/ca-bundle.trust.p11-kit (at least, on Fedora that's where it lives), so that's why my patch pulls in *that* bundle file.

Comment 22 Kai Engert (:kaie) (inactive account) 2017-05-10 13:35:14 UTC
Maybe it's not yet fully clear to you how libnssckbi.so and the replacement p11-kit-trust.so work.

The symlink named libnssckbi.so can alternatively point to either

(a)
the file libnssckbi.so provided by a nss.rpm package.

This is a statically operating module.
It doesn't read any other files.
It contains the set of CA trust as provided by Mozilla, both positive trust and distrust.

It ignores any local system configuration.

Your guess in comment 2 is wrong, libnssckbi.so doesn't access /etc/pki/nssdb/

It's an application decision if an application uses /etc/pki/nssdb/ as the NSS database init path.

I don't know how curl initializes NSS. If curl chooses to use the read-only NSS init APIs, which don't take any database parameter, then directory /etc/pki/nssdb/ isn't accessed.


If curl uses read-only NSS init APIs, and curl loads the module named libnssckbi.so, then having the nss.rpm package installed should be sufficient to give you the default trust, and allow the loading of libnssckbi.so to work.


(b)
p11-kit-trust.so from the p11-kit-trust.rpm package.

If this is installed, you need much more to operate correctly.

Please don't try to guess what files you should manually copy to make it work.

If p11-kit-trust is installed, you should make sure that the ca-certificates package is installed, too.

The ca-certificates package installs multiple files. There is no guarantee that any single file is sufficient for p11-kit-trust.so to work.

In order to get the intended behavior, you need the full set of files that the p11-kit-trust package installs.

p11-kit-trust.so will dynamically scan the /usr/share/pki/ca-trust-source and /etc/pki/ca-trust/source directories (both), to learn about the shipped CAs.

Some files inside thoes directories might be symlinks, which point to files outside these two directories.

This is used by the ca-legacy configuration, to potentially enable an additional set of CAs for backward compatibility.


The files 
  /etc/pki/tls/certs/ca-bundle.crt
  /etc/pki/tls/certs/ca-bundle.trust.crt
are *NOT* read by p11-kit-trust.so

Those are backwards compatibility filenames, that point to a converted and combined result of all trust information.

For its internal operation, sufficient for applications that load the p11-kit-trust.so module as their source of CA trust, it's sufficient for it to be able to read all the trust source directories (plus the result of navigating symbolic links inside those trust source directories).

For compatibility with applications that don't support loading p11-kit-trust.so, it's possible to run the utility update-ca-trust, which will call into p11-kit-trust.so, and ask it to produce the set union of all trust, and write it to different file formats, e.g. java keystore file format, and openssl plain certificate file format (ca-bundle.crt is a symlink to that), and openssl extended certificate file format that includes trust/distrust flags (ca-bundle-trust.crt is a symlink to that).

Comment 23 Kai Engert (:kaie) (inactive account) 2017-05-10 13:42:12 UTC
So, if your intention is to create an environment, where curl works to access https sites, and you don't need any local CA configuration in that environment, then it should be sufficient to pull in the libnssckbi.so file from the nss.rpm package. (Whatever way you use to obtain libnssckbi.so from nss.rpm, don't make it a one time copy that you check in somewhere. It should be a dynamic operation, to pull libnssckbi.so from the most recently published nss.rpm package.)

I believe libnssckbi.so doesn't need anything else from NSS, but you must have installed nspr.rpm

Comment 24 Kamil Dudka 2017-05-10 14:22:55 UTC
Thanks for the explanation, Kai!

In order to make it easier to maintain (although less secure), I propose to pick only %{libdir}/nss/libnssckbi.so and create %{libdir}/libnssckbi.so symlink to it.

Comment 25 Adam Williamson 2017-05-12 07:19:00 UTC
I talked it through in quite a lot of detail with Kai, and we agreed to an approach where we install whatever libnssckbi.so we find in a libdir (dracut already handles the problem of differing libdirs across distros) and then grep it for p11 trust directories and install any we find. I have updated the pull request to implement this.

We decided against just trying to always find and install the original NSS libnssckbi.so for a couple of reasons:

i) it *is* less configurable, and this is technically a regression from the previous behaviour - previously curl was using ca-bundle.crt , which respects local configuration of trust, so if we go from that to the NSS libnssckbi.so on a system where p11-kit-trust.so is available, we're going backwards

ii) we don't actually have any particularly reliable way to *find* the NSS libnssckbi.so (or even a cast iron guarantee it'll always be there); sure, right now on Fedora it's in libdir/nss , but what about other distros, if they also switch away from using a CA bundle file? what if it gets moved? We can search through all the subdirs of all libdirs with `find`, or something, but with a system like alternatives, it doesn't even really *have* to be under a libdir.

The PR has been updated, please do take a look and see what you think! Thanks.

Comment 26 Mike Ruckman 2017-05-12 21:12:31 UTC
Discussed in the 2017-05-02 Blocker Review meeting [0]. This is a violation of the following criterion: "The installer must be able to download and use an installer update image from an HTTP server" (note: 'HTTP' there is meant as 'HTTP not FTP or NFS', it does not mean 'we don't expect HTTPS to work'), and Accepted as a Blocker.

[0] Sorry for the late update on this.

Comment 27 Harald Hoyer 2017-05-15 12:49:13 UTC
dracut-044-181.fc26

Comment 28 Fedora Update System 2017-05-15 12:50:16 UTC
dracut-044-181.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-fa869268a2

Comment 29 Adam Williamson 2017-05-15 23:20:18 UTC
Dusty created a test image for anyone who wants to verify this fix:

https://dustymabe.fedorapeople.org/new-dracut.iso

if anyone else can check, the update needs one more +1. thanks!

Comment 30 Fedora Update System 2017-05-16 06:08:56 UTC
dracut-044-181.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-fa869268a2

Comment 31 Alessio 2017-05-16 08:58:34 UTC
(In reply to Adam Williamson from comment #29)
> https://dustymabe.fedorapeople.org/new-dracut.iso

It works.

In virt-manager, using Direct kernel boot, providing vmlinuz and initrd.img taken from this ISO, and using "console=ttyS0 inst.repo=https://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170513.n.0/compose/Workstation/x86_64/os" as kernel parameters I am able to successfully complete QA:Testcase Boot Methods Pxeboot (as stated before, it fails using vmlinuz and initrd.img from 20170513.n.0 compose).

Comment 32 Fedora Update System 2017-05-17 11:55:17 UTC
dracut-044-181.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.