Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1651348 - Armv7 guest fails on AArch64 - "cpu.c:906: arm_cpu_realizefn: Assertion no_aa32"
Summary: Armv7 guest fails on AArch64 - "cpu.c:906: arm_cpu_realizefn: Assertion no_aa32"
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: qemu
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Fedora Virtualization Maintainers
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: ARMTracker
TreeView+ depends on / blocked
 
Reported: 2018-11-19 19:08 UTC by Paul Whalen
Modified: 2018-12-12 12:50 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-12-12 12:50:31 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Paul Whalen 2018-11-19 19:08:23 UTC
Description of problem:
Attempting to boot an armv7 guest on an aarch64 host with aarch32 capabilities (Mustang xgene) fails with:

qemu-system-aarch64: /builddir/build/BUILD/qemu-3.1.0-rc1/target/arm/cpu.c:906: arm_cpu_realizefn: Assertion `no_aa32 || cpu_isar_feature(arm_div, cpu)' failed.
2018-11-19 17:04:36.735+0000: shutting down, reason=failed


Version-Release number of selected component (if applicable):
qemu-system-aarch64-3.1.0-0.1.rc1.fc30.aarch64

How reproducible:
Everytime.

Steps to Reproduce:

sudo virt-install \
    --name f30-armhfp --ram 4096 --arch armv7l --os-variant fedora22 \
    --disk /var/lib/libvirt/images/f30-armhfp.raw,bus=virtio,format=raw,size=8 \
    --location=https://kojipkgs.fedoraproject.org/compose//rawhide/Fedora-Rawhide-20181118.n.0/compose/Everything/armhfp/os/ \
    --extra-args="console=ttyAMA0 rw"

WARNING  Couldn't configure UEFI: Did not find any UEFI binary path for arch 'armv7l'
WARNING  Your VM may not boot successfully.

Starting install...
Retrieving file vmlinuz...                                                                                                                                             | 7.3 MB  00:00:01     
Retrieving file initrd.img...                                                                                                                                          |  54 MB  00:00:04     
Allocating 'f30-armhfp.raw'                                                                                                                                            | 8.0 GB  00:00:00     
ERROR    internal error: qemu unexpectedly closed the monitor: qemu-system-aarch64: /builddir/build/BUILD/qemu-3.1.0-rc1/target/arm/cpu.c:906: arm_cpu_realizefn: Assertion `no_aa32 || cpu_isar_feature(arm_div, cpu)' failed.

Comment 1 Jeremy Linton 2018-12-07 22:49:32 UTC
Well, I haven't hit this problem, although I have hit a bunch of other ones. I'm on a mustang now with kernel 4.20.0-0.rc5.git2, qemu 3.0.91 (qemu-3.1.0-0.1.rc1.fc30).

I'm on a b0 though, I wonder if that has something to do with it? There are a couple problems with the A3's (AFAIK, and I don't think fedora is carrying any of the workarounds). Let me see if I can reproduce it on an older mustang.

Comment 2 Jeremy Linton 2018-12-07 23:16:14 UTC
Actually, I have reproduced it on the mustang now.

Comment 3 Jeremy Linton (ARM) 2018-12-10 23:02:04 UTC
So, this is almost assuredly caused by https://github.com/qemu/qemu/commit/0f8d06f16c9d1041d728d09d464462ebe713c662.

Which is odd, as I've been struggling to understand how it works, given that on all the machines I've tested aa64pfr0_el1 is being trapped by the kernel and emulated to userspace, with the single detail that its _NOT_ indicating aarch32 support at any exception levels. That is because the kernel is sanitizing the feature registers, but because its not marked FTR_VISIBLE the default value being exported is aarch64 only support.

Comment 4 Richard Henderson 2018-12-11 17:06:55 UTC
Well, that isn't happening here:

(gdb) fin
Run till exit from #0  kvm_arm_get_host_cpu_features (
    ahcf=ahcf@entry=0xaaaaab804870 <arm_host_cpu_features>)
    at /home/rth/qemu/qemu/target/arm/kvm64.c:483
0x0000aaaaaaec76b8 in kvm_arm_set_cpu_features_from_host (cpu=0xffffbc529010)
    at /home/rth/qemu/qemu/target/arm/kvm.c:155
155	            return;
Value returned is $3 = true
(gdb) p/x arm_host_cpu_features.isar.id_aa64pfr0
$4 = 0x122

However, 

$ grep VERSION /etc/os-release 
VERSION="18.10 (Cosmic Cuttlefish)"
$ uname -a
Linux chuckanut 4.18.0-11-generic #12-Ubuntu SMP Tue Oct 23 19:24:51 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux
$ head -8 /proc/cpuinfo 
processor	: 0
BogoMIPS	: 100.00
Features	: fp asimd evtstrm cpuid
CPU implementer	: 0x50
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0x000
CPU revision	: 1

I'll see about trying a kernel newer than 4.18, and see if that's where the change came in.

We have missed the boat for fixing this for the qemu-3.1 release.  Which is annoying.
This should probably get an upstream launchpad bug report too.

Comment 5 Jeremy Linton 2018-12-11 21:32:06 UTC
Hmm (i'm not a qemu expert), so some of what this was based on was dumping the values from arm_cpu_realizefn, where it turns out ARM_FEATURE_AARCH64 isn't even set because -cpu host,aarch64=off disables it. So that code isn't even executing!

Hence when I saw that cpu_isar_feature equal to 0, I assumed that was because id_aa64pfr0 was 0x11, which is what you get if its read from userspace due to the HIDDEN/STRICT settings for the emulation. But dumping it, its also, so right before the assert() we have:

arm_feature(ARM_FEATURE_AARCH64)=0
cpu_isar_feature = 0
id_aa64pfr0=0

if dumped like:

"arm_feature(ARM_FEATURE_AARCH64)=%d cpu_isar_feature = %d , id_aa64pfr0=%lX\n",arm_feature(&cpu->env, ARM_FEATURE_AARCH64), cpu_isar_feature(aa64_aa32, cpu), cpu->isar.id_aa64pfr0

and run with qemu-system-aarch64 machine virt-3.1,accel=kvm -cpu host,aarch64=off

Comment 6 Richard Henderson 2018-12-11 21:57:20 UTC
Ok, knowing the command line options helps some,
but I still don't replicate either the assert or
what you're seeing.


(gdb) run -machine virt-3.1,accel=kvm -cpu host,aarch64=off
Thread 1 "qemu-system-aar" hit Breakpoint 2, arm_cpu_realizefn (
    dev=0xffffbc256010, errp=0xffffffffe670)
    at /home/rth/qemu/qemu/target/arm/cpu.c:906
906	        assert(no_aa32 || cpu_isar_feature(arm_div, cpu));
(gdb) p/x cpu->isar
$3 = {id_isar0 = 0x2101110, id_isar1 = 0x13112111, id_isar2 = 0x21232042, 
  id_isar3 = 0x1112131, id_isar4 = 0x10142, id_isar5 = 0x1, id_isar6 = 0x0, 
  mvfr0 = 0x10110222, mvfr1 = 0x12111111, mvfr2 = 0x43, id_aa64isar0 = 0x0, 
  id_aa64isar1 = 0x0, id_aa64pfr0 = 0x122, id_aa64pfr1 = 0x0}
(gdb) call isar_feature_arm_div(&cpu->isar)
$4 = true


The contents of id_aa64pfr0 should be completely ignored for aarch64=off.

I do still see kvm_arm_set_cpu_features_from_host being called in
order to grab the other id registers to fill in -cpu host.

Comment 7 Jeremy Linton 2018-12-11 22:20:55 UTC
Ok, I've been really confused about where id_aa6iasr0 has been comming from, and know I know whats going on (part of the clue was your line numbers/printing).

Upstream qemu head has a bunch of code to set the isar registers that (AFAIK) is missing from the release fedora is running. Looks like your commit was incompletely backported?

Comment 8 Richard Henderson 2018-12-11 22:25:46 UTC
OMG, now I feel foolish.  The clue was right here:

qemu-system-aarch64-3.1.0-0.1.rc1.fc30.aarch64
                              ^^^

The fix for the bug that you're seeing was included in 3.1.0-rc2.
The final 3.1.0 release came out today, fwiw.

Comment 9 Jeremy Linton 2018-12-11 23:14:43 UTC
I should have probably pulled qemu forward earlier to see if the problem exists. Anyway, it seems that its using KVM_GET_ONE_REG which returns the correct value for aa64pfr0 too.

Comment 10 Cole Robinson 2018-12-12 12:50:31 UTC
3.1.0 GA is built for rawhide now


Note You need to log in before you can comment on or make changes to this bug.