Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 2064115

Summary: Start encrypted tpm guest failed
Product: Red Hat Enterprise Linux 9 Reporter: Meina Li <meili>
Component: libvirtAssignee: Michal Privoznik <mprivozn>
libvirt sub component: General QA Contact: Yanqiu Zhang <yanqzhan>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: jdenemar, jsuchane, lmen, marcandre.lureau, mpitt, mprivozn, qcheng, virt-maint, weizhan, xuwei, xuzhang, yanqzhan
Version: 9.1Keywords: Automation, Regression, Triaged, Upstream
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-8.2.0-1.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-11-15 10:03:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Meina Li 2022-03-15 05:44:53 UTC
Description of problem:
Start encrypted tpm guest failed

Version-Release number of selected component (if applicable):
libvirt-8.1.0-1.el9.x86_64
qemu-kvm-6.2.0-11.el9.x86_64
swtpm-0.7.0-1.20211109gitb79fd91.el9.x86_64
libtpms-0.9.1-0.20211126git1ff6fe1f43.el9.x86_64
edk2-ovmf-20220126gitbb1bba3d77-3.el9.noarch
kernel-5.14.0-70.1.1.el9.x86_64
openssl-3.0.1-17.el9.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Prepare a tpm secret.
# cat secret.xml 
<secret ephemeral="no" private="yes">
<description>sample vTPM secret</description>
<usage type="vtpm">
<name>VTPM_example</name>
</usage>
</secret>
# virsh secret-define secret.xml 
Secret fe7c5949-528a-404b-9a63-2be66113796b created
# MYSECRET=`printf %s "open sesame" | base64`
# virsh secret-set-value --secret fe7c5949-528a-404b-9a63-2be66113796b $MYSECRET
error: Passing secret value as command-line argument is insecure!
Secret value set
# virsh secret-list
 UUID                                   Usage
-----------------------------------------------------------
 fe7c5949-528a-404b-9a63-2be66113796b   vtpm VTPM_example

2. Define a tpm guest with tpm device.
# vim lmn.xml
......
    <tpm model='tpm-crb'>
      <backend type='emulator' version='2.0'>
        <encryption secret='fe7c5949-528a-404b-9a63-2be66113796b'/>
      </backend>
    </tpm>
......
# virsh define lmn.xml 
Domain 'lmn' defined from lmn.xml

3.Start the guest.
# ls /var/lib/libvirt/swtpm/  --no tpm file
# virsh start lmn
error: Failed to start domain 'lmn'
error: internal error: qemu unexpectedly closed the monitor: 2022-03-15T05:28:05.862730Z qemu-kvm: tpm-emulator: TPM result for CMD_INIT: 0x101 operation failed

Actual results:
Start the guest failed

Expected results:
Can start guest

Additional info:
1) Swtpm.log:
Successfully created EK certificate locally.
Successfully created NVRAM area 0x1c00016 for ECC EK certificate.
Successfully activated PCR banks sha256 among sha1,sha256,sha384,sha512.
Successfully authored TPM state.
Ending vTPM manufacturing @ Tue 15 Mar 2022 06:28:05 AM CET
Verification of HMAC failed. Data integrity is compromised
SWTPM_NVRAM_LoadData: Error from SWTPM_NVRAM_GetDecryptedData rc = 33
Verification of HMAC failed. Data integrity is compromised
SWTPM_NVRAM_LoadData: Error from SWTPM_NVRAM_GetDecryptedData rc = 33
libtpms/tpm2: Entering failure mode; code: 8, location: NvPowerOn line 126
Error: Could not initialize libtpms.
Error: Could not initialize the TPM
Data client disconnected
2) After step 3, if we remove the tpm file and start again, the guest will start.
# rm -rf /var/lib/libvirt/swtpm/2108a219-bdf5-46b3-9ff2-acef47b48d23/tpm2/tpm2-00.permall 
# virsh start lmn
Domain 'lmn' started

Comment 6 Jaroslav Suchanek 2022-03-21 11:01:12 UTC
Michal, pls, have a look. Thanks.

Comment 9 Michal Privoznik 2022-03-21 15:03:05 UTC
Patch proposed on the list:

https://listman.redhat.com/archives/libvir-list/2022-March/229433.html

Comment 11 Yanqiu Zhang 2022-04-02 08:08:57 UTC
Issue is not reproduced on rhel9.1 with:
qemu-kvm-6.2.0-12.el9.x86_64
libvirt-8.2.0-1.el9.x86_64
swtpm-0.7.0-1.20211109gitb79fd91.el9.x86_64
libtpms-0.9.1-0.20211126git1ff6fe1f43.el9.x86_64
edk2-ovmf-20220126gitbb1bba3d77-4.el9.noarch
openssl-3.0.1-21.el9.x86_64

Steps:
# virsh start vm-ovmf 
Domain 'vm-ovmf' started

# virsh dumpxml vm-ovmf |grep /tpm -B5
    <tpm model='tpm-crb'>
      <backend type='emulator' version='2.0'>
        <encryption secret='40f4e01e-02d9-48e3-8b1d-b5985238d1e2'/>
      </backend>
      <alias name='tpm0'/>
    </tpm>

In guest os:
[root@localhost ~]# ls /dev/|grep tpm
tpm0
tpmrm0
[root@localhost ~]# tpm2_getrandom  --hex 16
d6837351b53a77315daee10a1414c784

Comment 13 Michal Privoznik 2022-04-04 06:56:01 UTC
Merged upstream as:

commit 4d7bb0177a33c4e90fd001edfe27bc030354d875
Author:     Michal Prívozník <mprivozn>
AuthorDate: Mon Mar 21 13:33:06 2022 +0100
Commit:     Michal Prívozník <mprivozn>
CommitDate: Mon Mar 28 10:00:18 2022 +0200

    qemu_tpm: Do async IO when starting swtpm emulator
    
    When vTPM is secured via virSecret libvirt passes the secret
    value via an FD when swtpm is started (arguments --key and
    --migration-key). The writing of the secret into the FDs is
    handled via virCommand, specifically qemu_tpm calls
    virCommandSetSendBuffer()) and then virCommandRunAsync() spawns a
    thread to handle writing into the FD via
    virCommandDoAsyncIOHelper. But the thread is not created unless
    VIR_EXEC_ASYNC_IO flag is set, which it isn't. In order to fix
    it, virCommandDoAsyncIO() must be called.
    
    The credit goes to Marc-André Lureau
    <marcandre.lureau> who has done all the debugging and
    proposed fix in the bugzilla.
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=2064115
    Fixes: a9c500d2b50c5c041a1bb6ae9724402cf1cec8fe
    Signed-off-by: Michal Privoznik <mprivozn>
    Reviewed-by: Jiri Denemark <jdenemar>

v8.1.0-229-g4d7bb0177a

While there are more fixes in the patchset, those are more cleanup of an internal code than bugfixes:

https://listman.redhat.com/archives/libvir-list/2022-March/229480.html

Comment 17 Yanqiu Zhang 2022-04-14 02:33:53 UTC
Auto regression test passed:
Pkgs info:
    libvirt	libvirt-8.2.0-1.el9.x86_64
    qemu-kvm	qemu-kvm-6.2.0-12.el9.x86_64
    kernel	kernel-5.14.0-75.el9.x86_64
    swtpm       0.7.0-1.20211109gitb79fd91.el9
    libtpms     0.9.1-0.20211126git1ff6fe1f43.el9
    edk2-ovmf   20220221gitb24306f15d-1.el9
Job url:
    https://libvirt-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/libvirt/view/RHEL-9.1%20x86_64/job/libvirt-RHEL-9.1-runtest-x86_64-function-tpm_emulator/6/testReport/
    All cases passed except 2 skipped by existing bz2025520.

Comment 19 errata-xmlrpc 2022-11-15 10:03:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Low: libvirt security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8003