Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 2237986

Summary: Fedora 39 fails to boot on qemu VM using OVMF_CODE_4M.secboot.qcow2 firmware
Product: [Fedora] Fedora Reporter: lnie <lnie>
Component: libvirtAssignee: Libvirt Maintainers <libvirt-maint>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 39CC: abologna, awilliam, berrange, clalancette, crobinso, geraldo.simiao.kutz, gmarr, jforbes, kparal, kraxel, laine, libvirt-maint, lruzicka, pbonzini, philip.wyett, philmd, robatino, virt-maint
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard: RejectedBlocker
Fixed In Version: libvirt-9.7.0-1.fc39 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-09-12 22:34:09 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
journal
none
screencast
none
screencast
none
screencast
none
VM settings at virt-manager
none
VM booted normally at Virt-manager
none
packages installed with the current iso
none
I installed libvirt for the libvirtd.service to run none

Description lnie 2023-09-08 09:06:26 UTC
Try to boot Fedora-Workstation-Live-x86_64-39-20230905.n.0.iso with virt-manager on f39 host,but failed,just "Display output is not active" on black screen is shown.

Reproducible: Always

Comment 1 lnie 2023-09-08 09:07:12 UTC
Created attachment 1987697 [details]
journal

Comment 2 lnie 2023-09-08 09:08:17 UTC
Created attachment 1987698 [details]
screencast

Comment 3 Fedora Blocker Bugs Application 2023-09-08 10:07:44 UTC
Proposed as a Blocker for 39-beta by Fedora user lnie using the blocker tracking app because:

 This violates:
The release must be able host virtual guest instances of the same release.

Comment 4 Adam Williamson 2023-09-08 21:02:51 UTC
This works for me, I've been testing this way for weeks. How is your VM configured? If you create a new VM, do you still have this problem? Thanks!

Comment 5 lnie 2023-09-09 06:51:40 UTC
Created attachment 1987801 [details]
screencast

Comment 6 lnie 2023-09-09 06:53:03 UTC
> How is your VM configured? 

Nothing special, just set it to uefi boot.

> If you create a new VM, do you still have this problem? Thanks!

yes

Comment 7 lnie 2023-09-09 07:00:05 UTC
Ah,I guess I found the cause,I set the Firmware to UEFI x86_64: /usr/share/edk2/ovmf/OVMF_CODE.secboot.fd.If I create a new VM with other Firmware,it works.
VMs with that fireware works well on f37 and f38 hosts,I guess that firmware need to be updated for f39,and this bug doesn't need to be a blocker?

Comment 8 Adam Williamson 2023-09-09 07:04:22 UTC
Ah, that's interesting - I actually saw the same thing a few days ago, but I'd never tried using that firmware setting before so I assumed it had been broken before. It's...a bit worrying if that firmware doesn't work, but we'd best ask an expert...let's re-assign to edk2 for now...

Comment 9 lnie 2023-09-09 07:24:11 UTC
> but I'd never tried using that firmware setting before so I assumed it had been broken before

I've been using it all the time,when I try to test UEFI back to BIOS time,and of course,now.

> let's re-assign to edk2 for now...

Thanks for doing the favor.

Comment 10 lnie 2023-09-09 07:33:59 UTC
> > but I'd never tried using that firmware setting before so I assumed it had been broken before
> I've been using it all the time,when I try to test UEFI back to BIOS time

That being said,it might had been broken before, not sure,I just test UEFI occasionally, back then:)

Comment 11 Cole Robinson 2023-09-10 15:54:21 UTC
My f39 host doesn't have a problem booting Fedora-Workstation-Live-x86_64-39-20230910.n.0.iso . New VM, choose ISO, set ram to 8096, disable storage, Customize Install, Select /usr/share/edk2/ovmf/OVMF_CODE.secboot.fd, and it boots to desktop

lnie is it 100% reproducible for you? does f38 workstation media fail too?

Comment 12 lnie 2023-09-11 07:40:12 UTC
> lnie is it 100% reproducible for you? does f38 workstation media fail too?

Yes,it's 100% reproducible,and f38 failed too.Actually,I found the bug on a beaker machine having f39 workstation system,
and then I reproduce it on my local machine,I know hardware are not related,just want to highlight that I see this bug Not only on one f39 host.

But when I tried with exactly what you said just now,I found what I'm using on f39 system is /usr/share/edk2/ovmf/OVMF_CODE_4M.secboot.qcow2(there is only /usr/share/edk2/ovmf/OVMF_CODE.secboot.fd on f38 host by default),sorry,I should check more carefully.

Comment 13 lnie 2023-09-11 07:42:07 UTC
Created attachment 1988089 [details]
screencast

Comment 14 lnie 2023-09-11 07:45:20 UTC
So this bug only happens,and 100% reproducible with /usr/share/edk2/ovmf/OVMF_CODE_4M.secboot.qcow2.

Comment 15 Lukas Ruzicka 2023-09-11 09:30:40 UTC
I have tested both OVMF_CODE_4M.secboot and OVMF_CODE.secboot.fd as a VM firmware on Fedora 39 fully updated. I cannot reproduce the bug, the VMs boot just fine not only into the LIVE mode, but also after installation. This holds true for both F39 Workstation image and Rawhide Workstation image.
I have not tested any other EFI modes yet, but if there is need, I can do it.

Comment 16 Gerd Hoffmann 2023-09-11 10:43:28 UTC
The mixed testing results suggest that there are some configuration problems ...

So, some background:

F38 has:
 (1) OVMF_CODE.secboot.fd + OVMF_VARS.secboot.fd.

F39 has:
 (1) OVMF_CODE.secboot.fd + OVMF_VARS.secboot.fd.
 (2) OVMF_CODE_4M.secboot.qcow2 + OVMF_VARS_4M.secboot.qcow2

Expected behavior:
 * Existing VMs (i.e. after F38 -> F39 upgrade) should continue to use (1).
 * Newly created VMs (virt-install --boot uefi) on F39 should use (2).

You can't freely mix and match code and vars, and when using the 4M qcow2
builds you also have to take care to specify the format.  libvirt xml
for (2) looks like this:

  <os firmware='efi'>
    <type arch='x86_64' machine='pc-q35-5.0'>hvm</type>
    <firmware>
      <feature enabled='yes' name='enrolled-keys'/>
      <feature enabled='yes' name='secure-boot'/>
    </firmware>
    <loader readonly='yes' secure='yes' type='pflash' format='qcow2'>/usr/share/edk2/ovmf/OVMF_CODE_4M.secboot.qcow2</loader>
    <nvram template='/usr/share/edk2/ovmf/OVMF_VARS_4M.secboot.qcow2' format='qcow2'>/var/lib/libvirt/qemu/nvram/fedora-org-base_VARS.qcow2</nvram>
  </os>

Comment 17 Geraldo Simião 2023-09-11 13:03:47 UTC
Testing here, cannot reproduce the bug, even using OVMF_CODE_4M.secboot.qcow2 on F39 workstation, VM worked fine.

Comment 18 Cole Robinson 2023-09-11 13:06:14 UTC
This could be libvirt version related. libvirt-9.7.0-1.fc39 is in updates-testing and is known to have some firmware autoselection fixes. I don't know what version I was using and I dismantled my setup. lnie are you on libvirt 9.6.0 ? does updating make a difference?

Comment 19 Geraldo Simião 2023-09-11 13:06:43 UTC
Created attachment 1988142 [details]
VM settings at virt-manager

Comment 20 Geraldo Simião 2023-09-11 13:07:59 UTC
Created attachment 1988153 [details]
VM booted normally at Virt-manager

on terminal is the host system version

Comment 21 Geraldo Simião 2023-09-11 13:11:46 UTC
here's the system is at 
libvirt-9.7.0-1.fc39.x86_64

Comment 22 Lukas Ruzicka 2023-09-11 14:09:14 UTC
I do not have the libvirt on my system like @geraldo.simiao.kutz. Instead I have 

python3-libvirt-9.7.0-1.fc39.x86_64

Comment 23 lnie 2023-09-11 14:58:43 UTC
> lnie are you on libvirt 9.6.0 ? does updating make a difference?
 Yes,and the problem is gone after I updated it to libvirt-9.7.0-1.fc39.x86_64,thanks

Comment 24 Geraldo Simião 2023-09-11 16:27:34 UTC
(In reply to Lukas Ruzicka from comment #22)
> I do not have the libvirt on my system like @geraldo.simiao.kutz.
> Instead I have 
> 
> python3-libvirt-9.7.0-1.fc39.x86_64

yeah, for libvirtd to run I must install the libvirt package, because the iso came without it.

Comment 25 Geraldo Simião 2023-09-11 16:29:22 UTC
Created attachment 1988180 [details]
packages installed with the current iso

Fedora-Workstation-Live-x86_64-39-20230910.n.0

Comment 26 Geraldo Simião 2023-09-11 16:30:51 UTC
Created attachment 1988181 [details]
I installed libvirt for the libvirtd.service to run

Because gnome boxes works fine but virt-manager need the libvirtd.servie runnning

Comment 27 Geraldo Simião 2023-09-11 16:33:06 UTC
and then I updated to new version on testing libvirt-9.7.0-1.fc39.x86_64

Comment 28 Adam Williamson 2023-09-11 17:25:32 UTC
The fixed libvirt is only in updates-testing, so the bug should not be closed.

Comment 29 Fedora Update System 2023-09-11 17:26:40 UTC
FEDORA-2023-57fd2e3393 has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-57fd2e3393

Comment 30 Geoffrey Marr 2023-09-11 18:24:44 UTC
Discussed during the 2023-09-11 blocker review meeting: [0]

The decision to classify this bug as a "RejectedBlocker (Beta)" was made as we don't believe GNOME Boxes offers the affected configuration, and anyone running into it with any other tool should have updates available at the time, so it's fine for this to go as a regular update.

[0] https://meetbot.fedoraproject.org/fedora-blocker-review/2023-09-11/f39-blocker-review.2023-09-11-16.00.txt

Comment 31 lnie 2023-09-12 03:15:47 UTC
>I do not have the libvirt on my system like @geraldo.simiao.kutz. Instead I have 

> python3-libvirt-9.7.0-1.fc39.x86_64

python-libvirt depends on the API provided by libvirt,I'm afraid you could not install python3-libvirt without libvirt installed.

> The fixed libvirt is only in updates-testing, so the bug should not be closed.

Ah,right,I was just trying to move this bug from the blocker list to save you guys'time last night,as it's fixed and clearly not a blocker.
So...,dear fedora expert:),next time,I should close the blocker issue ticket instead?

Comment 32 Adam Williamson 2023-09-12 06:24:15 UTC
As the person who proposed the bug as a blocker, you can "unpropose" it by clearing the blocker tracker from the Blocks: field. So, make it not block the Beta blocker tracker bug any more. We already rejected it at the meeting today, but in future you can do that :)

Comment 33 lnie 2023-09-12 13:39:49 UTC
> As the person who proposed the bug as a blocker, you can "unpropose" it by clearing the blocker tracker from the Blocks: field. So, make it not block the Beta blocker tracker bug any more. We already rejected it at >the meeting today, but in future you can do that :)

Thanks Adam,will try that way next time:)

Comment 34 Andrea Bolognani 2023-09-12 14:57:56 UTC
(In reply to Cole Robinson from comment #18)
> This could be libvirt version related. libvirt-9.7.0-1.fc39 is in
> updates-testing and is known to have some firmware autoselection fixes.

FWIW, I initially thought that the fixes made in 9.7.0 would not
affect this scenario; after looking more closely, however, I have
come to the conclusion that they absolutely do.


More specifically, when selecting the firmware in question
virt-manager will generate an XML that looks like

  <os>
    <type arch='x86_64' machine='q35'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/edk2/ovmf/OVMF_CODE_4M.secboot.qcow2</loader>
  </os>

which libvirt 9.6.0 interprets as

  <os>
    <type arch='x86_64' machine='q35'>hvm</type>
    <loader readonly='yes' type='pflash' format='raw'>/usr/share/edk2/ovmf/OVMF_CODE_4M.secboot.qcow2</loader>
  </os>

and, since the only firmware descriptor for that path is one that
says its format is qcow2, it can't find any matches:

  qemuFirmwareMatchDomain:1171 : No matching path in '/usr/share/qemu/firmware/41-edk2-ovmf-2m-raw-x64-sb.json'
  qemuFirmwareMatchDomain:1294 : Discarding loader with mismatching flash format 'qcow2' != 'raw'
  qemuFirmwareMatchDomain:1171 : No matching path in '/usr/share/qemu/firmware/50-edk2-ovmf-x64-microvm.json'

Notice how it skipped over

  /usr/share/qemu/firmware/50-edk2-ovmf-4m-qcow2-x64-nosb.json

which is the firmware descriptor we want.


This limitation has been addressed by

  commit 10a8997cbb402f7edb9f970af70feee2fc256a1c
  Author: Andrea Bolognani <abologna>
  Date:   Tue May 16 19:50:50 2023 +0200

    conf: Don't default to raw format for loader/NVRAM

    Due to the way the information is stored by the XML parser, we've
    had this quirk where specifying any information about the loader
    or NVRAM would implicitly set its format to raw. That is,

      <nvram>/path/to/guest_VARS.fd</nvram>

    would effectively be interpreted as

      <nvram format='raw'>/path/to/guest_VARS.fd</nvram>

    forcing the use of raw format firmware even when qcow2 format
    would normally be preferred based on the ordering of firmware
    descriptors. This behavior can be worked around in a number of
    ways, but it's fairly unintuitive.

    In order to remove this quirk, move the selection of the default
    firmware format from the parser down to the individual drivers.

    Most drivers only support raw firmware images, so they can
    unconditionally set the format early and be done with it; the
    QEMU driver, however, supports multiple formats and so in that
    case we want this default to be applied as late as possible,
    when we have already ruled out the possibility of using qcow2
    formatted firmware images.

    Signed-off-by: Andrea Bolognani <abologna>
    Reviewed-by: Michal Privoznik <mprivozn>

  https://gitlab.com/libvirt/libvirt/-/commit/10a8997cbb402f7edb9f970af70feee2fc256a1c

At the time I thought I was simply improving the user experience for
a fairly narrow corner case, but it's now clear to me that this
change is what allows the XML generated by virt-manager in this
scenario to be interpreted correctly. In retrospect, I sure am glad
that I didn't resist the urge to scratch this specific itch! O:-)

Comment 35 Andrea Bolognani 2023-09-12 15:15:46 UTC
(In reply to Geraldo Simião from comment #26)
> I installed libvirt for the libvirtd.service to run
> 
> Because gnome boxes works fine but virt-manager need the libvirtd.servie
> runnning

Note that it's technically not necessary to have the libvirt-daemon
package installed: Fedora uses modular daemons by default, and
virt-manager is able to connect directly to virtqemud without going
through the legacy monolithic daemon (libvirtd).

However, the message that is presented on startup is misleading:

  The libvirtd service doesn't appear to be installed. Install and
  run the libvirtd service to manage virtualization on this host.

  A virtualization connection can be manually added via File -> Add
  Connection.

This is not going to be a big problem in practice because libvirt
9.7.0 contains

  commit aa5895cbc72bd9b4bb1ce99e231b2ac4b25db9c4
  Author: Andrea Bolognani <abologna>
  Date:   Wed Aug 30 17:45:47 2023 +0200

    rpm: Recommend libvirt-daemon for with_modular_daemons distros

    A default deployment on modern distros uses modular daemons but
    switching back to the monolithic daemon, while not recommended,
    is still considered a perfectly valid option.

    For a monolithic daemon deployment, the upgrade to libvirt 9.2.0
    or newer works as expected; a subsequent call to dnf autoremove,
    however, results in the libvirt-daemon package being removed and
    the deployment no longer working.

    In order to avoid that situation, mark the libvirt-daemon as
    recommended.

    This will unfortunately result in it being included in most
    installations despite not being necessary, but considering that
    the alternative is breaking existing setups on upgrade it feels
    like a reasonable tradeoff.

    Moreover, since the dependency on libvirt-daemon is just a weak
    one, it's still possible for people looking to minimize the
    footprint of their installation to manually remove the package
    after installation, mitigating the drawbacks of this approach.

    https://bugzilla.redhat.com/show_bug.cgi?id=2232805

    Signed-off-by: Andrea Bolognani <abologna>
    Reviewed-by: Erik Skultety <eskultet>
    Reviewed-by: Daniel P. Berrangé <berrange>

  https://gitlab.com/libvirt/libvirt/-/commit/aa5895cbc72bd9b4bb1ce99e231b2ac4b25db9c4

which means that libvirt-daemon is going to be present on the system
unless the user manually uninstalled it.

That said, it would be nice if virt-manager didn't show a misleading
message in the first place. Cole, do you think it would be a lot of
work to make it detect virtqemud in addition to libvirtd?

Comment 36 Fedora Update System 2023-09-12 22:34:09 UTC
FEDORA-2023-57fd2e3393 has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 37 Cole Robinson 2023-09-24 20:33:59 UTC
(In reply to Andrea Bolognani from comment #35)
> (In reply to Geraldo Simião from comment #26)
> > I installed libvirt for the libvirtd.service to run
> > 
> > Because gnome boxes works fine but virt-manager need the libvirtd.servie
> > runnning
> 
> Note that it's technically not necessary to have the libvirt-daemon
> package installed: Fedora uses modular daemons by default, and
> virt-manager is able to connect directly to virtqemud without going
> through the legacy monolithic daemon (libvirtd).
> 
> However, the message that is presented on startup is misleading:
> 
>   The libvirtd service doesn't appear to be installed. Install and
>   run the libvirtd service to manage virtualization on this host.
> 
>   A virtualization connection can be manually added via File -> Add
>   Connection.
> 

Thanks for catching that.

A lot of the messaging virt-manager has around this is
out of date. I dropped some of the blocking checks and tried to
generalize the error messages:

https://github.com/virt-manager/virt-manager/commit/775edfd5dc668c26ffbdf07f6404ca80d91c3a3a

Comment 38 Andrea Bolognani 2023-09-25 12:32:40 UTC
(In reply to Cole Robinson from comment #37)
> (In reply to Andrea Bolognani from comment #35)
> > Note that it's technically not necessary to have the libvirt-daemon
> > package installed: Fedora uses modular daemons by default, and
> > virt-manager is able to connect directly to virtqemud without going
> > through the legacy monolithic daemon (libvirtd).
> > 
> > However, the message that is presented on startup is misleading:
> > 
> >   The libvirtd service doesn't appear to be installed. Install and
> >   run the libvirtd service to manage virtualization on this host.
> > 
> >   A virtualization connection can be manually added via File -> Add
> >   Connection.
> 
> Thanks for catching that.
> 
> A lot of the messaging virt-manager has around this is
> out of date. I dropped some of the blocking checks and tried to
> generalize the error messages:
> 
> https://github.com/virt-manager/virt-manager/commit/775edfd5dc668c26ffbdf07f6404ca80d91c3a3a

Looks good, thanks a lot Cole!