Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 2245707

Summary: [abrt] udisks2: __ioctl(): udisksd killed by SIGABRT
Product: [Fedora] Fedora Reporter: Camiropan <camiropan>
Component: udisks2Assignee: Tomáš Bžatek <tbzatek>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 39CC: awilliam, camiropan, gnome-sig, kparal, tbzatek, vtrefny
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
URL: https://retrace.fedoraproject.org/faf/reports/bthash/57b93d3d09bd1dfce7760e736bb1eecc9f7eb51
Whiteboard: abrt_hash:ca80b7352eeb9b35db2dcb31357b02f202b6a8f9;VARIANT_ID=workstation; AcceptedFreezeException
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-10-26 23:38:41 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2143447    
Attachments:
Description Flags
File: proc_pid_status
none
File: maps
none
File: limits
none
File: environ
none
File: open_fds
none
File: mountinfo
none
File: os_info
none
File: cpuinfo
none
File: core_backtrace
none
File: dso_list
none
File: var_log_messages
none
File: backtrace none

Description Camiropan 2023-10-23 17:45:10 UTC
Version-Release number of selected component:
udisks2-2.10.1-1.fc39

Additional info:
reporter:       libreport-2.17.11
type:           CCpp
reason:         udisksd killed by SIGABRT
journald_cursor: s=173419b830d34f46ac5374a2d7dda5e4;i=2327;b=ce4d163bb411426985dad39b4e03ddc8;m=2d384bb;t=6086569e7573b;x=e724155289bd6032
executable:     /usr/libexec/udisks2/udisksd
cmdline:        /usr/libexec/udisks2/udisksd
cgroup:         0::/system.slice/udisks2.service
rootdir:        /
uid:            0
kernel:         6.5.6-300.fc39.x86_64
package:        udisks2-2.10.1-1.fc39
runlevel:       N 5
backtrace_rating: 4
crash_function: __ioctl

Truncated backtrace:
Thread no. 1 (1 frames)
 #7 __ioctl at ../sysdeps/unix/sysv/linux/ioctl.c:43

Comment 1 Camiropan 2023-10-23 17:45:15 UTC
Created attachment 1995213 [details]
File: proc_pid_status

Comment 2 Camiropan 2023-10-23 17:45:17 UTC
Created attachment 1995214 [details]
File: maps

Comment 3 Camiropan 2023-10-23 17:45:20 UTC
Created attachment 1995215 [details]
File: limits

Comment 4 Camiropan 2023-10-23 17:45:23 UTC
Created attachment 1995216 [details]
File: environ

Comment 5 Camiropan 2023-10-23 17:45:26 UTC
Created attachment 1995217 [details]
File: open_fds

Comment 6 Camiropan 2023-10-23 17:45:28 UTC
Created attachment 1995218 [details]
File: mountinfo

Comment 7 Camiropan 2023-10-23 17:45:30 UTC
Created attachment 1995219 [details]
File: os_info

Comment 8 Camiropan 2023-10-23 17:45:31 UTC
Created attachment 1995220 [details]
File: cpuinfo

Comment 9 Camiropan 2023-10-23 17:45:33 UTC
Created attachment 1995221 [details]
File: core_backtrace

Comment 10 Camiropan 2023-10-23 17:45:34 UTC
Created attachment 1995222 [details]
File: dso_list

Comment 11 Camiropan 2023-10-23 17:45:35 UTC
Created attachment 1995223 [details]
File: var_log_messages

Comment 12 Camiropan 2023-10-23 17:45:37 UTC
Created attachment 1995224 [details]
File: backtrace

Comment 13 Fedora Update System 2023-10-23 19:02:11 UTC
FEDORA-2023-0f7d8165cb has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-0f7d8165cb

Comment 14 Tomáš Bžatek 2023-10-23 19:04:24 UTC
This is caused by a NVMe device not conforming to the official NVM Express specification. Contact your hardware vendor for firmware update.

Comment 15 Kamil Páral 2023-10-24 09:54:35 UTC
*** Bug 2245285 has been marked as a duplicate of this bug. ***

Comment 16 Kamil Páral 2023-10-24 09:58:48 UTC
Tomáš, will the new libblockdev work around the firmware bug?

If this is the case, let's propose this as a freeze exception, so that people can install F39 on these disks (you can read more details in bug 2245285 ).

Comment 17 Camiropan 2023-10-24 14:13:32 UTC
Unfortunately, there are no firmware updates for the SK hynix PC611 NVMe. Neither Dell nor SK hynix offer any kind of firmware updates.
I have installed Windows 10, dell update commander, and SK hynix Center, but as I said, there are no updates available.

Comment 18 Tomáš Bžatek 2023-10-24 18:25:59 UTC
(In reply to Kamil Páral from comment #16)
> Tomáš, will the new libblockdev work around the firmware bug?

Only partially. We've received number of similar reports upstream and worked with the NVMe community on a temporary solution. Still it's just a workaround for userspace until the kernel host nvme driver gets fixed (not much progress as of now).

Also needed are some libnvme patches (that I got upstreamed but haven't had time to make a Fedora backport). Will do that tomorrow.  

> If this is the case, let's propose this as a freeze exception, so that
> people can install F39 on these disks (you can read more details in bug
> 2245285 ).

Sure. The libblockdev update can go in independently from libnvme, no direct ABI dependencies.

Comment 19 Tomáš Bžatek 2023-10-24 19:31:43 UTC
So I have built libnvme-1.6-2.fc39 and libnvme-1.6-2.fc40

Please test both https://bodhi.fedoraproject.org/updates/FEDORA-2023-0f7d8165cb and https://bodhi.fedoraproject.org/updates/FEDORA-2023-b4ed0a0db1 together.

Comment 20 Kamil Páral 2023-10-25 08:12:54 UTC
(In reply to Tomáš Bžatek from comment #19)
> https://bodhi.fedoraproject.org/updates/FEDORA-2023-b4ed0a0db1 together.

Can you please mark the libnvme update as also fixing this bug? We need it for freeze exception tracking process. Thanks!

Comment 21 Kamil Páral 2023-10-25 09:52:05 UTC
@camiropan Can you please quickly test whether this F39 Workstation Live image works for you? It includes both the updates mentioned above:
https://fedorapeople.org/groups/qa/bugs/rhbz2245707.iso

Comment 22 Fedora Update System 2023-10-25 12:32:36 UTC
FEDORA-2023-b4ed0a0db1 has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-b4ed0a0db1

Comment 23 Camiropan 2023-10-25 13:14:43 UTC
I tested the Live image https://fedorapeople.org/groups/qa/bugs/rhbz2245707.iso and installed fedora 39 with it and everything works fine.
A small issue i noticed is that in the information tab in Gnome settings the capacity of the nvme is "unknown".
Besides that, everything works as expected.

So this patches of libblockdev and libnvme are a workaround in userspace for a problem in the kernel, which exists because of the SK hyndix firmware bug, right?

Comment 24 Kamil Páral 2023-10-25 13:40:24 UTC
Thanks for testing, Camiropan. Please give +1 karma to both Bodhi updates from comment 19. Thanks!

Comment 25 Tomáš Bžatek 2023-10-25 13:48:13 UTC
(In reply to Camiropan from comment #23)
> So this patches of libblockdev and libnvme are a workaround in userspace for
> a problem in the kernel, which exists because of the SK hyndix firmware bug,
> right?

Correct. This is not limited to SK Hynix, it's just one of the reputable manufacturers that often came up. We found evidence of some Phison-based drives as well (that got fixed by firmware update), though most issues were reported with random Chinese-made NVMe controllers.

> A small issue i noticed is that in the information tab in Gnome settings the
> capacity of the nvme is "unknown".

That's https://github.com/storaged-project/udisks/issues/1194, not all controllers apparently report Total NVM Capacity data, would need to find an algorithm to guess goood-enough size in the meantime.

Comment 26 Adam Williamson 2023-10-25 18:00:59 UTC
the libnvme patches are claimed to be "backports", but what are they backports *of*? I can find no trace that they've been submitted to or discussed at http://lists.infradead.org/pipermail/linux-nvme/ , which AFAICT is where they're supposed to go. The spec file does not provide any comment explaining their provenance (as it's supposed to).

We're trying to do a candidate compose and I'm really on the fence about whether it's a good idea to pull this stuff in.

Comment 27 Adam Williamson 2023-10-25 18:04:02 UTC
oh, wait - I see the libnvme repo at least has pull requests enabled. These appear to be:

* https://github.com/linux-nvme/libnvme/pull/727
* https://github.com/linux-nvme/libnvme/pull/731

The note that the second "received only minimal testing" seems a bit concerning. Both seem to have been merged with fairly cursory review.

Comment 28 Adam Williamson 2023-10-25 18:07:50 UTC
+4 in https://pagure.io/fedora-qa/blocker-review/issue/1420 , marking accepted. I am a bit worried this is quite a lot of change to land in a fairly critical component at this point, though. It does seem like Arch has had the changes landed for a week or two and no new bugs seem to have been reported.

Comment 29 Fedora Update System 2023-10-26 23:38:41 UTC
FEDORA-2023-0f7d8165cb has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 30 Fedora Update System 2023-10-26 23:38:54 UTC
FEDORA-2023-b4ed0a0db1 has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.