I've been told that aarch64's "virt" machinetype does support the dmi-to-pci-bridge. Can you attach the XML sent to libvirt?

Works fine for me with:

libguestfs-1.32.4-1.el7.aarch64
libvirt-1.2.17-13.el7_2.4.aarch64
qemu-kvm-rhev-2.5.0-4.el7.aarch64

Upgrading just libvirt to:

libvirt-1.3.4-1.el7.aarch64

causes the failure as described (still with qemu 2.5.0).

The XML is:

<?xml version="1.0"?>
<domain type="kvm" xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0">
  <name>guestfs-o4n2n8jim3j2zovy</name>
  <memory unit="MiB">768</memory>
  <currentMemory unit="MiB">768</currentMemory>
  <cpu mode="host-passthrough">
    <model fallback="allow"/>
  </cpu>
  <vcpu>1</vcpu>
  <clock offset="utc">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
  </clock>
  <os>
    <type machine="virt">hvm</type>
    <loader readonly="yes" type="pflash">/usr/share/AAVMF/AAVMF_CODE.fd</loader>
    <nvram>/tmp/libguestfsgvmESi/vars.fd.1</nvram>
    <kernel>/var/tmp/.guestfs-1000/appliance.d/kernel</kernel>
    <initrd>/var/tmp/.guestfs-1000/appliance.d/initrd</initrd>
    <cmdline>panic=1 console=ttyAMA0 earlyprintk=pl011,0x9000000 ignore_loglevel efi-rtc=noprobe udevtimeout=6000 udev.event-timeout=6000 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 guestfs_verbose=1 TERM=xterm-256color</cmdline>
  </os>
  <on_reboot>destroy</on_reboot>
  <devices>
    <controller type="scsi" index="0" model="virtio-scsi"/>
    <disk device="disk" type="file">
      <source file="/var/tmp/test1.img"/>
      <target dev="sda" bus="scsi"/>
      <driver name="qemu" type="raw" cache="writeback"/>
      <address type="drive" controller="0" bus="0" target="0" unit="0"/>
    </disk>
    <disk type="file" device="disk">
      <source file="/tmp/libguestfsgvmESi/overlay2"/>
      <target dev="sdb" bus="scsi"/>
      <driver name="qemu" type="qcow2" cache="unsafe"/>
      <address type="drive" controller="0" bus="0" target="1" unit="0"/>
      <shareable/>
    </disk>
    <serial type="unix">
      <source mode="connect" path="/tmp/libguestfsgvmESi/console.sock"/>
      <target port="0"/>
    </serial>
    <channel type="unix">
      <source mode="connect" path="/tmp/libguestfsgvmESi/guestfsd.sock"/>
      <target type="virtio" name="org.libguestfs.channel.0"/>
    </channel>
  </devices>
  <qemu:commandline>
    <qemu:env name="TMPDIR" value="/var/tmp"/>
  </qemu:commandline>
</domain>

(In reply to Laine Stump from comment #1)
> I've been told that aarch64's "virt" machinetype does support the
> dmi-to-pci-bridge.

It does, upstream and in Fedora. The i82801b11-bridge device
is disabled in aarch64 RHEL's QEMU, though, hence the error.

Ideally we'd come up with a PCI(e) topology that gives us
all the features we want without having to resort to stuff
derived from Intel hardware, such as i82801b11-bridge, on
aarch64.

(In reply to Andrea Bolognani from comment #3)

> It does, upstream and in Fedora. The i82801b11-bridge device
> is disabled in aarch64 RHEL's QEMU, though, hence the error.

> Ideally we'd come up with a PCI(e) topology that gives us
> all the features we want without having to resort to stuff
> derived from Intel hardware, such as i82801b11-bridge, on
> aarch64.

Agreed, but such devices don't exist right now, so the road to getting that solution is much longer than the one where you just re-enable the Intel devices. (Unless you're willing to use this as an issue to push on getting generic PCI controllers into qemu, which is definitely a "Good Thing(tm)".

In the meantime, having no dmi-to-pci-bridge is going to make for a *horrible* experience with libvirt, since it currently wants to auto-assign all devices to legacy PCI slots (since pretty much none of them are PCIe devices). Also, someone upstream enabled PCI on aarch64 by copying what is done for Q35, meaning that libvirt will always try to add a dmi-to-pci-bridge to the domain config.

(In reply to Laine Stump from comment #4)
> > It does, upstream and in Fedora. The i82801b11-bridge device
> > is disabled in aarch64 RHEL's QEMU, though, hence the error.
> 
> > Ideally we'd come up with a PCI(e) topology that gives us
> > all the features we want without having to resort to stuff
> > derived from Intel hardware, such as i82801b11-bridge, on
> > aarch64.
> 
> Agreed, but such devices don't exist right now, so the road to getting that
> solution is much longer than the one where you just re-enable the Intel
> devices. (Unless you're willing to use this as an issue to push on getting
> generic PCI controllers into qemu, which is definitely a "Good Thing(tm)".

The problem with enabling the Intel-based gizmos is that,
even if we plan it as a stop gap measure, guests will end
up having references to them in their configuration and so
we'll be stuck having to provide them forever.

I don't think anyone wants that to happen unless it truly
is the only way to get to virtio-pci on aarch64.

> In the meantime, having no dmi-to-pci-bridge is going to make for a
> *horrible* experience with libvirt

Right now, unless you switch to a custom QEMU binary, or
stick to an old libvirt version, you are simply unable to
create *any* guest on aarch64. That's horrible alright :)

> since it currently wants to auto-assign
> all devices to legacy PCI slots (since pretty much none of them are PCIe
> devices).

virtio devices can be PCIe nowadays (QEMU 2.6), as long as
you add disable-modern=off and disable-legacy=on when you
specify them.

Granted, there's no support for that in libvirt yet, but it
should be possible to convince libvirt to assign PCIe devices
to PCIe slots instead of PCI slots, right?

> Also, someone upstream enabled PCI on aarch64 by copying what is
> done for Q35, meaning that libvirt will always try to add a
> dmi-to-pci-bridge to the domain config.

Yeah, that was a mistake. That's why we're trying to figure
out a reasonable alternative topology for mach-virt, possibly
one that doesn't have PCI at all, or at least uses PCI
controllers that can actually be found on aarch64 hardware.

(In reply to Andrea Bolognani from comment #5)
> > since it currently wants to auto-assign
> > all devices to legacy PCI slots (since pretty much none of them are PCIe
> > devices).
> 
> virtio devices can be PCIe nowadays (QEMU 2.6), as long as
> you add disable-modern=off and disable-legacy=on when you
> specify them.

That is incredibly clunky, and scheduled to change (at least the disable-modern will have its default set to off in the next qemu release). I think it's a bad idea to permanently clutter up libvirt with something that will only be needed for a very short time (afaik, there will be no way to tell what is the default).

> 
> Granted, there's no support for that in libvirt yet, but it
> should be possible to convince libvirt to assign PCIe devices
> to PCIe slots instead of PCI slots, right?

Sure, it will already do that, if the CONNECT_TYPE for the device is set to PCIE (in qemuDomainCollectPCIAddress()).

> 
> we're trying to figure
> out a reasonable alternative topology for mach-virt, possibly
> one that doesn't have PCI at all, or at least uses PCI
> controllers that can actually be found on aarch64 hardware.

Keep in mind that if you want hotplug, even if youo have no legacy PCI devices, you will still need at least a pcie-root-port (only implementation in qemu now is ioh3420 - an Intel device) (and you'll need pcie-switch-(upstream|downstream)-port if you have more devices than will fit in the root complex; those are also currently only available in qemu via an intel-specific device - the xio3130).

I agree that making generic versions of these controllers available is the right thing to do long term. I don't know that it can be done in a reasonable time for RHEL 7.3, though.

Here are the 4 possible solutions:

1) postpone to RHEL 7.4
2) re-enable intel PCIe controllers
3) forbid hotplug and tell libvirt to assign everything directly to root complex (pcie-root).
4) get qemu to implement generic PCIe controllers.

In the meantime, hopefully the PCIe-ness of the virtio controllers will become usable in a reasonable manner; libvirt will need to become aware of that.

I think the *technically* best solution would be to forbid legacy PCI devices on arm, and create generic pcie-*-port devices in qemu (but not for the pci-to-dmi-bridge device). I don't know how realistic it is though.

This has been solved upstream as of the following commit:

commit 86a68bdb0ceddd7813ac7c0f47b07ecc029a5159
Author: Andrea Bolognani <abologna>
Date:   Thu Jun 16 19:13:19 2016 +0200

    qemu: Permit PCI-free aarch64 mach-virt guests

    There has been some progress lately in enabling virtio-pci on
    aarch64 guests; however, guest OS support is still spotty at best,
    so most guests are going to be using virtio-mmio instead.

    Currently, mach-virt guests are closely modeled after q35 guests,
    and that includes always adding a dmi-to-pci-bridge that's just
    impossible to get rid of. While that's acceptable (if suboptimal)
    for q35, where you will always need some kind of PCI device anyway,
    mach-virt guests should be allowed to avoid it.

v1.3.5-314-g86a68bd

It would be nice to have a scratch build of what is proposed
here so I can do a full integration test of libvirt + libguestfs
+ kernel changes on RHELSA.

Verified with packages:
libguestfs-1.32.5-9.el7.aarch64
qemu-kvm-rhev-2.6.0-11.el7.aarch64
libvirt-2.0.0-1.el7.aarch64

Verify steps:
1. Prepare an aarch64 host.
2. # guestfish -N fs:ext3


It finished successfully.

So verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2577.html