Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1688283
Summary: | Clevis fails to unlock encrypted partition with iot (ostree) installation | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Paul Whalen <pwhalen> | ||||||
Component: | clevis | Assignee: | Sergio Correia <scorreia> | ||||||
Status: | CLOSED CURRENTRELEASE | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||
Severity: | unspecified | Docs Contact: | |||||||
Priority: | unspecified | ||||||||
Version: | 31 | CC: | fmartine, nicolasoliver03, npmccallum, pbrobinson, pk.eskse, scorreia, sztsian, zsun | ||||||
Target Milestone: | --- | Keywords: | Reopened | ||||||
Target Release: | --- | ||||||||
Hardware: | Unspecified | ||||||||
OS: | Linux | ||||||||
Whiteboard: | |||||||||
Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||
Doc Text: |
If this bug requires documentation, please select an appropriate Doc Type value.http://iot.stg.fedoraproject.org/
|
Story Points: | --- | ||||||
Clone Of: | Environment: | ||||||||
Last Closed: | 2020-09-08 14:53:46 UTC | Type: | Bug | ||||||
Regression: | --- | Mount Type: | --- | ||||||
Documentation: | --- | CRM: | |||||||
Verified Versions: | Category: | --- | |||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||
Embargoed: | |||||||||
Bug Depends On: | |||||||||
Bug Blocks: | 1269538 | ||||||||
Attachments: |
|
Description
Paul Whalen
2019-03-13 12:58:13 UTC
After booting the system to check: [root@fitlet2 ~]# luksmeta show -d /dev/sda3 0 active empty 1 active cb6e8904-81ff-40da-a84a-07ab9ab5715e 2 inactive empty 3 inactive empty 4 inactive empty 5 inactive empty 6 inactive empty 7 inactive empty Password is correct: [root@fitlet2 ~]# cryptsetup luksOpen --test-passphrase --key-slot 0 /dev/sda3 && echo correct Enter passphrase for /dev/sda3: correct Remove the password and try again: [root@fitlet2 ~]# clevis luks unbind -d /dev/sda3 -s 1 The unbind operation will wipe a slot. This operation is unrecoverable. Do you wish to erase LUKS slot 1 on /dev/sda3? [ynYN] y Enter any remaining passphrase: [root@fitlet2 ~]# luksmeta show -d /dev/sda3 0 active empty 1 inactive empty 2 inactive empty 3 inactive empty 4 inactive empty 5 inactive empty 6 inactive empty 7 inactive empty Try to use the clevis commands again: [root@fitlet2 ~]# UUID=$(lsblk | grep luks | sed 's/^.*luks-//' | cut -d ' ' -f1) [root@fitlet2 ~]# DEV=$(blkid --uuid $UUID) [root@fitlet2 ~]# echo $DEV /dev/sda3 [root@fitlet2 ~]# clevis luks bind -f -k- -d $DEV tpm2 '{}' <<< fedora [root@fitlet2 ~]# luksmeta show -d /dev/sda3 0 active empty 1 active cb6e8904-81ff-40da-a84a-07ab9ab5715e 2 inactive empty 3 inactive empty 4 inactive empty 5 inactive empty 6 inactive empty 7 inactive empty Reboot and it again fails to unlock. The same commands work on a standard F29/30 installation. (In reply to Paul Whalen from comment #0) [snip] > > Works when using a standard Fedora 30 installation with the same package > versions and kickstart snippets. > > Clevis is included in the initramfs for IoT: > > lsinitrd initramfs-5.0.0-0.rc8.git0.1.fc30.x86_64.img | grep clevis > clevis Are the tpm2-tools and libraries also included in the initramfs? i.e: lsinitrd /boot/initramfs-$(uname -r).img | grep -E 'clevis|tpm2|tss2' lsinitrd initramfs-5.0.0-300.fc30.x86_64.img | grep -E 'clevis|tpm2|tss2' clevis -rw-r--r-- 1 root root 1 Feb 14 14:03 etc/cmdline.d/99clevis.conf -rwxr-xr-x 1 root root 1563 Aug 13 2018 usr/bin/clevis -rwxr-xr-x 1 root root 1614 Aug 13 2018 usr/bin/clevis-decrypt -rwxr-xr-x 1 root root 33304 Jan 31 09:46 usr/bin/clevis-decrypt-sss -rwxr-xr-x 1 root root 2721 Aug 13 2018 usr/bin/clevis-decrypt-tang -rwxr-xr-x 1 root root 3794 Aug 13 2018 usr/bin/clevis-decrypt-tpm2 -rwxr-xr-x 1 root root 95408 Feb 3 04:10 usr/bin/tpm2_createprimary -rwxr-xr-x 1 root root 92712 Feb 3 04:10 usr/bin/tpm2_load -rwxr-xr-x 1 root root 87968 Feb 3 04:10 usr/bin/tpm2_pcrlist -rwxr-xr-x 1 root root 108424 Feb 3 04:10 usr/bin/tpm2_unseal -rwxr-xr-x 1 root root 303880 Feb 14 14:03 usr/lib64/libtss2-mu.so.0.0.0 lrwxrwxrwx 1 root root 19 Feb 14 14:03 usr/lib64/libtss2-mu.so.0 -> libtss2-mu.so.0.0.0 -rwxr-xr-x 1 root root 403216 Feb 14 14:03 usr/lib64/libtss2-sys.so.0.0.0 lrwxrwxrwx 1 root root 20 Feb 14 14:03 usr/lib64/libtss2-sys.so.0 -> libtss2-sys.so.0.0.0 -rwxr-xr-x 1 root root 36416 Feb 14 14:03 usr/lib64/libtss2-tcti-device.so.0.0.0 lrwxrwxrwx 1 root root 28 Feb 14 14:03 usr/lib64/libtss2-tcti-device.so.0 -> libtss2-tcti-device.so.0.0.0 -rwxr-xr-x 1 root root 49 Jan 31 09:46 usr/lib/dracut/hooks/initqueue/online/60-clevis-hook.sh -rwxr-xr-x 1 root root 49 Jan 31 09:46 usr/lib/dracut/hooks/initqueue/settled/60-clevis-hook.sh -rwxr-xr-x 1 root root 2873 Aug 13 2018 usr/libexec/clevis-luks-askpass With Fedora-IoT-ostree-x86_64-30-20190215.0.iso I get: # echo foobar | clevis encrypt tpm2 '{}' | clevis decrypt WARNING:tcti:src/tss2-tcti/tcti-device.c:254:tcti_device_receive() Got EOF instead of response. ERROR: Create Object Failed ! ErrorCode: 0xa0008 ERROR: Unable to run tpm2_create Creating TPM2 object for jwk failed! # tpm2_rc_decode 0xa0008 error layer hex: 0xa0000 identifier: TSS2_TCTI_RC_LAYER description: Error from the TCTI base error code identifier: TSS2_BASE_RC_NO_CONNECTION description: Fails to connect to next lower layer # uname -r 5.0.0-0.rc6.git1.1.fc30.x86_64 Which is reported to be a regression in the kernel: https://github.com/tpm2-software/tpm2-tools/issues/1356 Following is the thread in the linux-integrity mailing list where the problem and possible solutions are being discussed: https://www.spinics.net/lists/linux-integrity/msg06386.html Upstream patch that solves this bug: https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1959715.html kernel-tools-5.0.4-300.fc30 kernel-headers-5.0.4-300.fc30 kernel-5.0.4-300.fc30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-4e7590b99c kernel-tools-5.0.4-200.fc29 kernel-headers-5.0.4-200.fc29 kernel-5.0.4-200.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-0ba1e6642f kernel-5.0.4-300.fc30, kernel-headers-5.0.4-300.fc30, kernel-tools-5.0.4-300.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-4e7590b99c kernel-5.0.4-200.fc29, kernel-headers-5.0.4-200.fc29, kernel-tools-5.0.4-200.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-0ba1e6642f kernel-5.0.4-200.fc29, kernel-headers-5.0.4-200.fc29, kernel-tools-5.0.4-200.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report. kernel-5.0.4-300.fc30, kernel-headers-5.0.4-300.fc30, kernel-tools-5.0.4-300.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report. Using compose Fedora-IoT-30-20190327.0 [root@fitlet2 ~]# echo foo | clevis encrypt tpm2 '{}' | clevis decrypt foo [root@fitlet2 ~]# uname -r 5.0.0-300.fc30.x86_64 But the automatic decryption still fails on boot. If I add 'rd.break=initqueue' to the kernel args to get a shell prior to decryption, then 'exit' the system will decrypt the volume and boot to log in. It appears to be working to some degree but perhaps a timing issue? This also happens with a F29 compose which includes kernel-4.20.16-200.fc29. Created attachment 1555593 [details]
tpm2-luks1-ks.cfg
Created attachment 1555594 [details]
tpm2-luks2-ks.cfg
I tried to reproduce this issue on the same hardware than Paul used and couldn't reproduce the exact issue but had a different one that could be related. Using the kickstart tpm2-luks1-ks.cfg file attached in Comment 14, the system was installed correctly but the LUKS volume was not unlocked automatically by clevis. Looking at the LUKSv1 and LUKSMeta headers, I noticed that a LUKSMeta slot was added for the bound pin but no a LUKS slot: $ sudo luksmeta show -d /dev/sda3 | grep "^[0-9][ ]*active" 0 active empty 1 active cb6e8904-81ff-40da-a84a-07ab9ab5715e $ sudo cryptsetup luksDump /dev/sda3 | grep ENABLED Key Slot 0: ENABLED But if I bind a pin to a LUKS volume after the installation, it did work correctly: $ sudo luksmeta wipe -d /dev/sda3 -s 1 $ sudo clevis luks bind -d /dev/sda3 tpm2 '{}' $ sudo luksmeta show -d /dev/sda3 | grep "^[0-9][ ]*active" 0 active empty 1 active cb6e8904-81ff-40da-a84a-07ab9ab5715e $ sudo cryptsetup luksDump /dev/sda3 | grep ENABLED Key Slot 0: ENABLED Key Slot 1: ENABLED And also the LUKS volume was automatically unlocked on boot. It also worked correctly if I used LUKSv2 instead of LUKSv1 like in the tpm2-luks2-ks.cfg kickstart file attached in Comment 15. So the problem seems to be only when storing the clevis pin on a LUKSv1 volume and when the clevis luks bind command is executed in the kickstart file. I've been having a similar issue on Fedora Silverblue, as this seems to still be open I thought I'd post it here rather than open a new bug report. It seems to be that the required initramfs modules are not loaded by the layer provided by what I assume would be the "clevis-dracut" package (if that's not the correct terminology let me know as I only installed Silverblue yesterday to play around with). Enabling client side generation of the initramfs via "rpm-ostree initramfs --enable" resolves this. (In reply to Javier Martinez Canillas from comment #16) > I tried to reproduce this issue on the same hardware than Paul used and > couldn't reproduce the exact issue but had a different one that could be > related. > > Using the kickstart tpm2-luks1-ks.cfg file attached in Comment 14, the > system was installed correctly but the LUKS volume was not unlocked > automatically by clevis. Looking at the LUKSv1 and LUKSMeta headers, I > noticed that a LUKSMeta slot was added for the bound pin but no a LUKS slot: > > $ sudo luksmeta show -d /dev/sda3 | grep "^[0-9][ ]*active" > 0 active empty > 1 active cb6e8904-81ff-40da-a84a-07ab9ab5715e > > $ sudo cryptsetup luksDump /dev/sda3 | grep ENABLED > Key Slot 0: ENABLED > > But if I bind a pin to a LUKS volume after the installation, it did work > correctly: > > $ sudo luksmeta wipe -d /dev/sda3 -s 1 > > $ sudo clevis luks bind -d /dev/sda3 tpm2 '{}' > > $ sudo luksmeta show -d /dev/sda3 | grep "^[0-9][ ]*active" > 0 active empty > 1 active cb6e8904-81ff-40da-a84a-07ab9ab5715e > > $ sudo cryptsetup luksDump /dev/sda3 | grep ENABLED > Key Slot 0: ENABLED > Key Slot 1: ENABLED > > And also the LUKS volume was automatically unlocked on boot. > > It also worked correctly if I used LUKSv2 instead of LUKSv1 like in the > tpm2-luks2-ks.cfg kickstart file attached in Comment 15. > > So the problem seems to be only when storing the clevis pin on a LUKSv1 > volume and when the clevis luks bind command is executed in the kickstart > file. Javier: Can you still reproduce this issue? (In reply to Sergio Correia from comment #19) > (In reply to Javier Martinez Canillas from comment #16) [snip] > > > > So the problem seems to be only when storing the clevis pin on a LUKSv1 > > volume and when the clevis luks bind command is executed in the kickstart > > file. > > Javier: Can you still reproduce this issue? I don't have access to the machine where I was able to reproduce the issue. But you could ask Paul Whalen or Peter Robinson if they are still facing issues with clevis and TPM2 devices in the Fedora IoT spin. This message is a reminder that Fedora 30 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora 30 on 2020-05-26. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '30'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 30 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. I have a similar issue happen earlier this year. I believe it can be the same (or at least similar) cause. Description of my problem: Given I have my home directory on luks volume with Network-Bound Disk Encryption configured (clevis + tang, tang is on RHEL-7) on my laptop. Earlier this year after updating a big couple of software in my system (for both my RHEL-7 and my laptop), I can no longer unlock my disk. I cannot really debug a lot since the tang server is in the office, while we are not allowed to go back to the office at the moment. Today I got my laptop from office with the help of a colleague. I tried to downgrade either clevis or cryptsetup, and also booting into older kernels, system cannot unlock my home partition during boot even if I typed the right password. From the log on the screen, I find something suspicious May 06 18:13:50 systemd[1]: Created slice system-dbus\x2d:1.2\x2dorg.freedesktop.problems.slice. May 06 18:13:50 systemd[1]: Started dbus-:1.2-org.freedesktop.problems. May 06 18:13:50 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=dbus-:1.2-org.freedesktop.problems@0 comm="systemd" exe="/usr/l> May 06 18:13:50 systemd-cryptsetup[5839]: Failed to activate with specified passphrase. (Passphrase incorrect?) May 06 18:13:50 systemd-tty-ask-password-agent[5841]: Failed to query password: Connection refused May 06 18:13:50 systemd-tty-ask-password-agent[5841]: Failed to show password: Connection refused May 06 18:13:50 systemd[1]: Starting Clevis LUKS systemd-ask-password Responder... May 06 18:13:50 abrt-notification[5965]: Process 487 (plymouthd) crashed in _nl_load_domain.cold() May 06 18:13:50 clevis-luks-askpass[5966]: Error communicating with the server! May 06 18:13:50 systemd-cryptsetup[5839]: Set cipher aes, mode xts-plain64, key size 256 bits for device /dev/fedora/home. May 06 18:13:50 systemd[1]: clevis-luks-askpass.service: Succeeded. May 06 18:13:50 systemd[1]: Started Clevis LUKS systemd-ask-password Responder. May 06 18:13:50 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=clevis-luks-askpass comm="systemd" exe="/usr/lib/systemd/system> May 06 18:13:50 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=clevis-luks-askpass comm="systemd" exe="/usr/lib/systemd/systemd> May 06 18:13:53 systemd-cryptsetup[5839]: Failed to activate with specified passphrase. (Passphrase incorrect?) May 06 18:13:53 systemd-cryptsetup[5839]: Too many attempts to activate; giving up. May 06 18:13:53 systemd[1]: systemd-cryptsetup: Main process exited, code=exited, status=1/FAILURE May 06 18:13:53 systemd[1]: systemd-cryptsetup: Failed with result 'exit-code'. May 06 18:13:53 systemd[1]: Failed to start Cryptography Setup for home. May 06 18:13:53 systemd[1]: Dependency failed for /dev/mapper/home. May 06 18:13:53 systemd[1]: Dependency failed for File System Check on /dev/mapper/home. May 06 18:13:53 systemd[1]: Dependency failed for /home. May 06 18:13:53 systemd[1]: Dependency failed for Remote File Systems. The log do shows failed to activate with specified password at least twice, however I only type the password once before it shows too many attempts. Workaround: I tried to boot without my /home and then totally remove clevis-dracut and clevis-systemd, dracut -f. Then I can unlock it with password after reboot. Steps of my setup is mostly here https://bugzilla.redhat.com/show_bug.cgi?id=1574413#c0 Additional info: $ sudo cryptsetup luksDump /dev/fedora/home | egrep -i "enable|version" Version: 1 Key Slot 0: ENABLED Key Slot 1: ENABLED $ sudo luksmeta show -d /dev/fedora/home 0 active empty 1 active cb6e8904-81ff-40da-a84a-07ab9ab5715e 2 inactive empty 3 inactive empty 4 inactive empty 5 inactive empty 6 inactive empty 7 inactive empty $ rpm -q clevis dracut cryptsetup clevis-12-1.fc31.x86_64 dracut-050-26.git20200316.fc31.x86_64 cryptsetup-2.3.0-1.fc31.x86_64 $ uname -r 5.6.8-200.fc31.x86_64 I'd like to help during my spare time if you need some more information and if it is not destructive. But again since the tang server is in the office, I cannot help test the tang related part. Hi Zamir, (In reply to Zamir SUN from comment #22) > I have a similar issue happen earlier this year. I believe it can be the > same (or at least similar) cause. > [...] > From the log on the screen, I find something suspicious > > May 06 18:13:50 systemd[1]: Created slice > system-dbus\x2d:1.2\x2dorg.freedesktop.problems.slice. > May 06 18:13:50 systemd[1]: Started > dbus-:1.2-org.freedesktop.problems. > May 06 18:13:50 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 > ses=4294967295 subj=system_u:system_r:init_t:s0 > msg='unit=dbus-:1.2-org.freedesktop.problems@0 comm="systemd" exe="/usr/l> > May 06 18:13:50 systemd-cryptsetup[5839]: Failed to activate with specified > passphrase. (Passphrase incorrect?) This looks suspicious. Are you able to unlock the device with say clevis luks unlock -d <device>? Paul: would you be able to check whether the issue reported here is still reproducible with clevis-14? Hi Sergio, This issue seemed to be the specific hardware I was using, but I am happy to report it is now working in Fedora IoT. We also have an automated test in OpenQA with swtpm which is also working. Closing as fixed. Thanks! I think I am running into this same issue in Red Hat 8.3 Edge. The encrypted partitions I set up during kickstart installation are not automatically unlocked by the clevis-luks-askpass@ service. But if I execute the binding again, once the OS is deployed, then the service can automatically unlock them. My kickstart is something like this %post --erroronfail --log=/root/disk-encryption-configuration.log set -euxo pipefail lvcreate -y -l 20%VG -n opt system printf "password" | cryptsetup -q luksFormat /dev/system/opt -d - printf "password" | clevis luks bind -f -k- -d /dev/system/opt tpm2 '{}' printf "password" | cryptsetup luksOpen /dev/system/opt c1 -d - mkfs.ext4 /dev/mapper/c1 sleep 1 cryptsetup luksClose c1 echo "luks-system-opt /dev/system/opt none _netdev" >> /etc/crypttab echo "/dev/mapper/luks-system-opt /var/opt ext4 defaults,_netdev 0 2" >> /etc/fstab %end When the service tries to unlock it, it complains about a TPM integrity policy. And after I execute this line again, once the OS is deployed, printf "password" | clevis luks bind -f -k- -d /dev/system/opt tpm2 '{}' Then the service can unlock it during boot. I guess it tries the first slot and fails, and then succeed with the second slot. I am using rhel-8.3-beta-1-x86_64-boot.iso to install. Should this fix be included in that boot iso? (In reply to nicolasoliver03 from comment #26) > I think I am running into this same issue in Red Hat 8.3 Edge. > [...] > When the service tries to unlock it, it complains about a TPM integrity > policy. > And after I execute this line again, once the OS is deployed, This seems to be a different issue. Would you mind opening a new BZ for tracking this? Done, see bug 1877470 Thanks Sergio! |