Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 2026747
| Summary: | I/O errors on f34 s390x kvm guests under f35 hypervisor | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Kevin Fenzi <kevin> | ||||||
| Component: | qemu | Assignee: | Fedora Virtualization Maintainers <virt-maint> | ||||||
| Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||
| Severity: | unspecified | Docs Contact: | |||||||
| Priority: | unspecified | ||||||||
| Version: | 35 | CC: | berrange, cfergeau, crobinso, ondrejj, pampelmuse, pbonzini, philmd, rjones, smitterl, virt-maint | ||||||
| Target Milestone: | --- | ||||||||
| Target Release: | --- | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | qemu-6.1.0-14.fc35 | Doc Type: | If docs needed, set a value | ||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2022-02-15 01:37:44 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
|
Description
Kevin Fenzi
2021-11-25 17:51:28 UTC
If you're using libvirt, please provide the libvirt XML config and the corresponding QEMU log file /var/log/libvirt/qemu/$GUEST.log. If using QEMU direct, just the full QEMU command line you use. Created attachment 1843618 [details]
libvirt xml for one of the guests
Here's the libvirt xml for a guest.
I did also try changing between the s390-ccw-virtio-6.1 (what f35 virt-install puts there) and s390-ccw-virtio-4.1 (what the f34 virt-install has there) and that didn't seem to matter.
Created attachment 1843619 [details]
log for guest
Here's the /var/log/libvirt/qemu/buildvm-s390x-23.s390.fedoraproject.org.log
Hi Kevin! We've seen a similar problem in downstream RHEL ... The fix is likely https://gitlab.com/qemu-project/qemu/-/commit/cc071629539dc1f303175a7 ... do you happen to know how to rebuild the package to have a try whether that fixes the issue for you? (In reply to Thomas Huth from comment #6) > Hi Kevin! We've seen a similar problem in downstream RHEL ... The fix is > likely https://gitlab.com/qemu-project/qemu/-/commit/cc071629539dc1f303175a7 > ... do you happen to know how to rebuild the package to have a try whether > that fixes the issue for you? Agreed. A) Could you try with qemu 6.2-rc to see if it still reproduces? B) Can you confirm a) the device attached to target 'vda' is backed by virtio-blk and a dasd block device (/dev/dasdX)? b) if so, with qemu 6.1 version you mention this should reproduce only easily reproduced without nested: 1. attach disk /dev/dasdX as virtio-blk to L1 guest 2. inside L1 guest format and create filesystem ext4 3. read write, check logs for I/O errors Here is a scrath build with the mentioned upstream fix: https://koji.fedoraproject.org/koji/taskinfo?taskID=79265813 if you confirm whether that solves the I/O errors, I'll do a formal build I know everyone's talking about s390x and dasd's and such, but is it possible this same bug could affect armv7 guest on aarch64 host (Fedora Rawhide in both cases)? For about a month I've been seeing errors like this inside the guest, and they look very similar to this BZ. Also Dan if there's a scratch building including aarch64 I could test it. [16784.102801] blk_update_request: I/O error, dev vda, sector 9689856 op 0x1:(WRITE) flags 0x800 phys_seg 16 prio class 0 [16784.130038] blk_update_request: I/O error, dev vda, sector 9690368 op 0x1:(WRITE) flags 0x4800 phys_seg 80 prio class 0 [16784.134209] blk_update_request: I/O error, dev vda, sector 9692928 op 0x1:(WRITE) flags 0x4800 phys_seg 80 prio class 0 [16784.138263] blk_update_request: I/O error, dev vda, sector 9695488 op 0x1:(WRITE) flags 0x800 phys_seg 96 prio class 0 [16784.142771] blk_update_request: I/O error, dev vda, sector 9696752 op 0x1:(WRITE) flags 0x4800 phys_seg 205 prio class 0 [16784.146855] blk_update_request: I/O error, dev vda, sector 9699312 op 0x1:(WRITE) flags 0x800 phys_seg 52 prio class 0 [16784.150877] blk_update_request: I/O error, dev vda, sector 9700136 op 0x1:(WRITE) flags 0x4800 phys_seg 157 prio class 0 [16784.155033] blk_update_request: I/O error, dev vda, sector 9702696 op 0x1:(WRITE) flags 0x4800 phys_seg 80 prio class 0 [16784.159096] blk_update_request: I/O error, dev vda, sector 9705256 op 0x1:(WRITE) flags 0x800 phys_seg 19 prio class 0 [16784.163163] blk_update_request: I/O error, dev vda, sector 9705864 op 0x1:(WRITE) flags 0x4800 phys_seg 80 prio class 0 [16784.188541] vda3: writeback error on inode 4750392, offset 43192320, sector 9682176 (In reply to Richard W.M. Jones from comment #9) > I know everyone's talking about s390x and dasd's and such, but is it > possible this same bug could affect armv7 guest on aarch64 host (Fedora > Rawhide in both cases)? For about a month I've been seeing errors > like this inside the guest, and they look very similar to this BZ. Which was filed as: https://bugzilla.redhat.com/show_bug.cgi?id=2004120 Scratch build with the patch, on top of Rawhide, all arches: https://koji.fedoraproject.org/koji/taskinfo?taskID=79266649 Can confirm it fixes the armv7/aarch64 problem here. *** Bug 2004120 has been marked as a duplicate of this bug. *** Build for Rawhide: https://koji.fedoraproject.org/koji/taskinfo?taskID=79268037 Build for F35: https://koji.fedoraproject.org/koji/taskinfo?taskID=79268098 (In reply to Richard W.M. Jones from comment #14) > Build for F35: > https://koji.fedoraproject.org/koji/taskinfo?taskID=79268098 The GCC / systemtap / armv7 bug of doom is now affecting F35 too. New F35 build including my workaround for that bug: https://koji.fedoraproject.org/koji/taskinfo?taskID=79268858 FEDORA-2021-12f6c46ad8 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2021-12f6c46ad8 FEDORA-2021-12f6c46ad8 has been pushed to the Fedora 35 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-12f6c46ad8` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-12f6c46ad8 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates. I've upgraded to that version of qemu and restarted a guest. Will see how it looks in a while. :) Thanks for the quick fixes. So far that guest is fine. :) So, looks good here... Seems to be a duplicate of this bug: https://bugzilla.redhat.com/show_bug.cgi?id=2041227 *** Bug 2041227 has been marked as a duplicate of this bug. *** FEDORA-2022-3a60c34473 has been pushed to the Fedora 35 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-3a60c34473` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-3a60c34473 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates. FEDORA-2022-3a60c34473 has been pushed to the Fedora 35 stable repository. If problem still persists, please make note of it in this bug report. |