Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1449837

Summary: Unable to create aarch64 images with virt-manager due to missing gicv3 emulation
Product: [Fedora] Fedora Reporter: Jeremy Linton <jeremy.linton>
Component: libvirtAssignee: Libvirt Maintainers <libvirt-maint>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 26CC: abologna, agedosier, amit, berrange, cfergeau, clalancette, crobinso, dwmw2, eric.auger, itamar, laine, libvirt-maint, pbonzini, pbrobinson, phrdina, rjones, veillard, virt-maint
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-3.2.1-3.fc26 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1450433 (view as bug list) Environment:
Last Closed: 2017-06-09 19:19:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 245418    

Description Jeremy Linton 2017-05-10 20:58:15 UTC
Description of problem: qemu/aarch64 only supports gicv3 emulation via KVM on a aarch64 machine. This wouldn't be a problem except that the last couple of qemu releases default to gicv3, which means that the default machine config created via libvirt/virt-manager fails with a message "MSIX is not supported by interrupt controller" (older versions say) "MSI is not supported by interrupt controller".


Version-Release number of selected component (if applicable): qemu 2.8.11 & 2.9


How reproducible: 100%


Steps to Reproduce:
1. fedpkg install qemu libvirt virt-manager
2. virt-manager
3. new machine, arch = aarch64, type=virt
3. Install from fedora aarch64.iso
4. next a couple times, until finish.
5. Failure with above message.

Actual results:
"MSIX is not supported by interrupt controller"

Expected results:
Emulated machine starts up, and starts install fedora 

Additional info:
This could be a libvirt type bug as well, because the gic type can be overriden with `gic_version=2` to make it work as well.


A quick hack in virt.c to create_its() would return a failure code, which when handled in create_gic() would flip the gic type=2, and proceed to create_v2m().

Comment 1 Jeremy Linton 2017-05-10 20:58:48 UTC
This bug is likely in F26 as well, I just haven't verified it there.

Comment 2 Cole Robinson 2017-05-10 21:05:03 UTC
Andrea I think I saw some internal discussion about this but I'm light on details. Can you explain the plan here?

Comment 3 Andrea Bolognani 2017-05-11 15:51:27 UTC
(In reply to Cole Robinson from comment #2)
> Andrea I think I saw some internal discussion about this but I'm light on
> details. Can you explain the plan here?

As detailed in Bug 1414081, the root of the issue is
that QEMU doesn't fully emulate GICv3, and more
specifically is missing the MSI controller which is
a requirement for virtio-pci.

There are several ways we could deal with the issue:

  1) switch back from virtio-pci to virtio-mmio, which
     works with the current GICv3;

  2) implement the missing bits of GICv3 emulation;

  3) change libvirt to always prefer GICv2 for TCG
     guests;

  4) provide a way for virt-manager users to choose
     between GICv2 and GICv3.

1) would be a massive step back, so I've included it
only for completeness' sake and strongly advise against
even considering it.

2) is something that we want to have at some point,
hence Bug 1414081, but I also have to assume it would
be a lot of work which rules it out as a short to medium
term solution.

3) is probably our best bet. We'll want to have a way
for libvirt to figure out whether QEMU supports full
GICv3 emulation so we can switch the default back to
that in the future.

4) is something that we should arguably have in any
case, same way you can already pass gic_version= to
virt-install in order to override libvirt's default.

Comment 4 Eric Auger 2017-05-12 07:11:20 UTC
Actually virtio-pci does not require any MSI controller. It can work with legacy INTx interrupts.

However the new PCIe topology requires it (the ioh3420 root port requires it and this causes the failure).

So I don't think 1) or reverting to the former topology is the right direction.
I confirm 2) is a really huge work.

3) looks the easiest (GICv2m MSI controller then will be used). Then 4)

Comment 5 Andrea Bolognani 2017-05-12 07:49:59 UTC
(In reply to Eric Auger from comment #4)
> Actually virtio-pci does not require any MSI controller. It can work with
> legacy INTx interrupts.
> 
> However the new PCIe topology requires it (the ioh3420 root port requires it
> and this causes the failure).

Sorry for being inaccurate :)

At the end of the day the distinction is irrelevant to us
though, because we only care about PCIe anyway.

> So I don't think 1) or reverting to the former topology is the right
> direction.
> I confirm 2) is a really huge work.
> 3) looks the easiest (GICv2m MSI controller then will be used). Then 4)

3) and 4) should be tackled in parallel.

Comment 6 Cole Robinson 2017-05-17 16:35:02 UTC
Fixed in libvirt.git, needs these commits:

commit 5645badd1fe04fee7237c2f95e7710e978e40770
Author: Andrea Bolognani <abologna>
Date:   Fri May 12 14:38:08 2017 +0200

    gic: Remove VIR_GIC_VERSION_DEFAULT
    
    The QEMU default is GICv2, and some of the code in libvirt
    relies on the exact value. Stop pretending that's not the
    case and use GICv2 explicitly where needed.
    
    Signed-off-by: Andrea Bolognani <abologna>

commit bc07101a7c2cd2ce07ad1ca28c47e0a7cde5625d
Author: Andrea Bolognani <abologna>
Date:   Fri May 12 13:29:57 2017 +0200

    qemu: Use GICv2 for aarch64/virt TCG guests
    
    There are currently some limitations in the emulated GICv3
    that make it unsuitable as a default. Use GICv2 instead.
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1450433
    
    Signed-off-by: Andrea Bolognani <abologna>

commit b24eaf6210ebaf5dc8d29621063873c8419c517e
Author: Andrea Bolognani <abologna>
Date:   Fri May 12 14:05:55 2017 +0200

    tests: Check default GIC version for aarch64/virt TCG guests
    
    Signed-off-by: Andrea Bolognani <abologna>

commit 5290d4fdaf7758a03b06d60d113993babee0a9d5
Author: Andrea Bolognani <abologna>
Date:   Fri May 12 11:03:19 2017 +0200

    qemu: Use qemuDomainMachineIsVirt() more
    
    Signed-off-by: Andrea Bolognani <abologna>

Comment 7 Andrea Bolognani 2017-05-18 07:06:43 UTC
Do you not plan to implent a way for users to switch
between GIC versions from the GUI then? virt-install
already allows you to do that.

Comment 8 Cole Robinson 2017-05-18 11:04:26 UTC
(In reply to Andrea Bolognani from comment #7)
> Do you not plan to implent a way for users to switch
> between GIC versions from the GUI then? virt-install
> already allows you to do that.

Yeah I can do that

Comment 10 Cole Robinson 2017-05-18 13:43:34 UTC
(In reply to Pavel Hrdina from comment #9)
> https://www.redhat.com/archives/virt-tools-list/2016-June/msg00056.html

Whoops, forgot about that :)

But re-reading my emails I think the reasoning for not exposing this in the UI still applies. Basically users will always want the latest gic version that the config supports, and changing it to something older would only be for testing purposes or maybe some obscure migration compat case. It's just that our understanding of a working config missed the tcg+virtio-pci case so libvirt's default was wrong. That's fixed, so all is good.

Yes having clicky UI would have given a virt-manager workaround here but a UI element who's primary use is bug workaround isn't very compelling to me.

Comment 11 Pavel Hrdina 2017-05-19 07:59:05 UTC
Np :)

These kind of UI elements could be hidden under "Advanced configuration" which could be enabled/disabled in the "Preferences" and by default it would be disabled to make the UI simple for the majority of users.

Comment 12 Fedora Update System 2017-05-30 23:37:24 UTC
libvirt-3.2.1-2.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-1bfce27ae1

Comment 13 Fedora Update System 2017-06-01 03:19:14 UTC
libvirt-3.2.1-2.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-1bfce27ae1

Comment 14 Fedora Update System 2017-06-09 19:19:13 UTC
libvirt-3.2.1-3.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.