Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1608242
| Summary: | CONFIG_CRYPTO_DEV_SP_PSP=y causes udev issues on AMD AGESA 1.0.0.4c+ | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Steven Haigh <netwiz> |
| Component: | kernel | Assignee: | Kernel Maintainer List <kernel-maint> |
| Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 28 | CC: | airlied, alexander, ben.r.xiao, bskeggs, chunnayya, darkbasic, equeim, ewk, faber, hdegoede, ichavero, itamar, jackos1998, jadit2, jappleii, jarodwilson, jcline, jeremy, jglisse, johannespfau, john.j5live, jonathan, josef, kernel-maint, kmazurek, labbott, linville, lmmason25, lnykryn, mati865, mchehab, mjg59, msekleta, netwiz, ng0177, oyvinds, pasik, paul.howell+bugzilla, redhat, ryan, ssahani, s, s.so, steved, steve, systemd-maint, Tim, timo, tobias.mueller, vedran, wpoely86, zbyszek |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | kernel-4.18.12-300.fc29 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-10-08 19:03:28 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
The impact of this is: 1) For an installed system, a ~2 minute wait in booting, then a normal boot. 2) For a live image, the image fails to boot. Have tried the following live images: * kde F28 1.1 * kde F28 updated spin 20180722. Possibly related from journalctl: Jul 25 15:11:43 systemd-udevd[598]: seq 3552 '/devices/system/cpu/cpu0' is taking a long time Jul 25 15:11:43 systemd-udevd[598]: seq 3383 '/devices/pci0000:00/0000:00:07.1/0000:1d:00.2' is taking a long time Jul 25 15:11:43 systemd-udevd[598]: seq 3567 '/devices/system/cpu/cpu9' is taking a long time Jul 25 15:11:43 systemd-udevd[598]: seq 3566 '/devices/system/cpu/cpu8' is taking a long time Jul 25 15:11:43 systemd-udevd[598]: seq 3553 '/devices/system/cpu/cpu1' is taking a long time Jul 25 15:11:43 systemd-udevd[598]: seq 3565 '/devices/system/cpu/cpu7' is taking a long time Jul 25 15:11:43 systemd-udevd[598]: seq 3564 '/devices/system/cpu/cpu6' is taking a long time Jul 25 15:11:43 systemd-udevd[598]: seq 3563 '/devices/system/cpu/cpu5' is taking a long time Jul 25 15:11:43 systemd-udevd[598]: seq 3562 '/devices/system/cpu/cpu4' is taking a long time Jul 25 15:11:43 systemd-udevd[598]: seq 3561 '/devices/system/cpu/cpu3' is taking a long time Jul 25 15:11:43 systemd-udevd[598]: seq 3560 '/devices/system/cpu/cpu2' is taking a long time Jul 25 15:11:43 systemd-udevd[598]: seq 3555 '/devices/system/cpu/cpu11' is taking a long time Jul 25 15:11:43 systemd-udevd[598]: seq 3558 '/devices/system/cpu/cpu14' is taking a long time Jul 25 15:11:43 systemd-udevd[598]: seq 3557 '/devices/system/cpu/cpu13' is taking a long time Jul 25 15:11:43 systemd-udevd[598]: seq 3556 '/devices/system/cpu/cpu12' is taking a long time Jul 25 15:11:43 systemd-udevd[598]: seq 3559 '/devices/system/cpu/cpu15' is taking a long time Jul 25 15:11:43 systemd-udevd[598]: seq 3554 '/devices/system/cpu/cpu10' is taking a long time Jul 25 15:12:43 systemd[1]: systemd-udev-settle.service: Main process exited, code=exited, status=1/FAILURE Jul 25 15:12:43 systemd[1]: systemd-udev-settle.service: Failed with result 'exit-code'. Jul 25 15:12:43 systemd[1]: Failed to start udev Wait for Complete Device Initialization. I notice a PCI bus address, this may help? # lspci 00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex 00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge 00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59) 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51) 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 03:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43d0 (rev 01) 03:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] Device 43c8 (rev 01) 03:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43c6 (rev 01) 16:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43c7 (rev 01) 16:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43c7 (rev 01) 16:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43c7 (rev 01) 16:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43c7 (rev 01) 16:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43c7 (rev 01) 18:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) 1c:00.0 VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] (rev a1) 1c:00.1 Audio device: NVIDIA Corporation GP106 High Definition Audio Controller (rev a1) 1d:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 145a 1d:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor 1d:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] USB 3.0 Host controller 1e:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 1455 1e:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) 1e:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller I have a feeling this is a BIOS issue. If I use BIOS version 230, this issue occurs. Downgrading to version 220 and the problem goes away. Ryzen/ThreadRipper? I have the same issue on Gigabyte X399 with UEFI F10, if I go back to F3j it works fine. Here's one more example with ASRock X470: http://forum.asrock.com/forum_posts.asp?TID=9179&title=new-asrock-x470-taichi-uefi-150 Another system described in bug 1613153. Good find. Yes, the CPU in question is a Ryzen 2700x. I have a case open with MSI regarding the BIOS update - as v23 fails, but v22 works perfectly - very similar to the forum thread. As such, its seeming likely that its a change in the 1.0.0.4 AGESA code. MSI include 1.0.0.4C in BIOS v23. I've also tried pinging a contact at AMD to see if I can get the issue raised to the correct area or get some feedback. Just received word that my message has been forwarded to the AMD BIOS team. Hopefully I can get a reply from them. *** Bug 1614773 has been marked as a duplicate of this bug. *** (In reply to Vedran Miletić from comment #4) > Ryzen/ThreadRipper? I have the same issue on Gigabyte X399 with UEFI F10, if > I go back to F3j it works fine. > > Here's one more example with ASRock X470: > http://forum.asrock.com/forum_posts.asp?TID=9179&title=new-asrock-x470- > taichi-uefi-150 That's exactly the issue I've been having on the Gigabyte X399. I'll flash the BIOS back from F10 to F3j and see if it goes away. However, in the meantime, if I mask systemd-udev-settle, everything seems to work fine without it. So far. I encountered something similar to this running Gigabyte X399 F10 BIOS. Have you tried doing sysrq-w to see what actually is blocking on udev? (Just to confirm if this is same issue or not) My blocking stack looked like this: [ 131.619989] sysrq: SysRq : Show Blocked State [ 131.620016] task PC stack pid father [ 131.620086] systemd-udevd D12624 7505 6593 0x80000120 [ 131.620121] Call Trace: [ 131.620143] ? __schedule+0x35b/0x730 [ 131.620169] schedule+0x37/0x90 [ 131.620195] __sev_do_cmd_locked+0xb3/0x130 [ccp] [ 131.620233] ? wait_woken+0x80/0x80 [ 131.620260] __sev_platform_init_locked+0x26/0x40 [ccp] [ 131.620296] sev_platform_init+0x18/0x30 [ccp] [ 131.620328] psp_pci_init+0x2b/0xb0 [ccp] [ 131.620356] ? driver_register+0x7b/0xc0 [ 131.620387] sp_mod_init+0x11/0x1000 [ccp] [ 131.620413] ? 0xffffffffc09b6000 [ 131.620435] do_one_initcall+0x35/0x1c0 [ 131.620461] ? free_unref_page_commit.isra.90+0x5f/0xc0 [ 131.620494] ? kmem_cache_alloc_trace+0x167/0x1d0 [ 131.620526] do_init_module+0x56/0x1ef [ 131.620551] load_module+0x20be/0x2970 [ 131.620578] ? kernel_read_file+0x16d/0x190 [ 131.620607] ? __do_sys_finit_module+0xa2/0xb0 [ 131.620634] __do_sys_finit_module+0xa2/0xb0 [ 131.620663] do_syscall_64+0x43/0xf0 In my case a simple workaround was to build a kernel with CONFIG_CRYPTO_DEV_SP_PSP disabled (which causes any SEV calls to return -ENODEV). (In reply to Vedran Miletić from comment #4) > Ryzen/ThreadRipper? I have the same issue on Gigabyte X399 with UEFI F10, if > I go back to F3j it works fine. > > Here's one more example with ASRock X470: > http://forum.asrock.com/forum_posts.asp?TID=9179&title=new-asrock-x470- > taichi-uefi-150 I rolled back to F3g from F3j, and my system started playing happily again. I upgraded to the Gigabute X399 BIOS version F3j (AGESA 1.0.0.5) When I dropped back to the previous BIOS level F3g (AGESA 1.0.0.3 Patch 4), everything started working again and udev settle worked again, mapping was happy, and even the kvm_amd error I was seeing here https://bugzilla.redhat.com/show_bug.cgi?id=1612608 was fixed by downgrading the BIOS. I've passed this info on - hopefully it makes its way to the BIOS team at AMD as well. It seems AGESA 1.0.0.4c onwards has issues right now. I guess its up to each vendor as to when they push a new BIOS after an updated AGESA from AMD... I have received a reply from AMD: "The support team has not seen this issue yet. You’ll need to work with the OEM so that the OEM can use the support channels to work with our support team on debugging and resolving the issue." As such, I would suggest sending this issue to Gigabyte in your case. I have a ticket open with MSI for my board, but still fighting through level 1 support at the moment. It has also been reported that newer AGESA versions break TR4 in the same way. A few people have mentioned that if CONFIG_CRYPTO_DEV_SP_PSP is set to 'n', or if build as a module, it is blocked from loading, that this issue does not present itself. Doesn't assist with Fedora however, as: $ grep CONFIG_CRYPTO_DEV_SP_PSP /boot/config-4.17.14-200.fc28.x86_64 CONFIG_CRYPTO_DEV_SP_PSP=y As such, this may be a problem in the PSP driver, or AGESA. (In reply to Steven Haigh from comment #15) > It has also been reported that newer AGESA versions break TR4 in the same > way. > > A few people have mentioned that if CONFIG_CRYPTO_DEV_SP_PSP is set to 'n', > or if build as a module, it is blocked from loading, that this issue does > not present itself. > > Doesn't assist with Fedora however, as: > $ grep CONFIG_CRYPTO_DEV_SP_PSP /boot/config-4.17.14-200.fc28.x86_64 > CONFIG_CRYPTO_DEV_SP_PSP=y > > As such, this may be a problem in the PSP driver, or AGESA. Note that the module that needs to be blocked is ccp. This, however, breaks kvm_amd as it uses symbols from ccp module. On kernel, https://github.com/torvalds/linux/blob/06dd3dfeea60e2a6457a6aedf97afc8e6d2ba497/drivers/crypto/ccp/sp-dev.c#L275-L277 are the lines affected by the option. Good news, I have received a reply from AMD regarding the issue after submitting further information as gathered above and from other sources. Quote: It would appear that the BIOS/firmware is advertising it supports SEV, when in fact it doesn't. We currently don't have a timeout associated with the SEV commands and so the module load is stuck - which would also explain the KVM issue. The current approach is to add a timeout to the kernel that stops things sticking forever and submit that to the current stable releases. I have also asked if this could be forwarded on to the BIOS team to not advertise SEV on hardware that doesn't support it - I thought this could be an OEM thing, but it might be unlikely that it is multiple OEMs making the same mistake. After hours of searching, I'm here to say I also have this issue. System is AMD Ryzen x2700 on Gigabyte x470 running the latest BIOS F4 which has the new AGESA 1.0.0.4. The Linux booting issue happened just after I updated the BIOS. I first noticed LVM taking a long time to process, and finally identified the below error message with systemctl status systemd-udevd.service: Aug 20 21:22:03 localhost.localdomain systemd-udevd[744]: worker [774] terminated by signal 9 (KILL) Aug 20 21:22:03 localhost.localdomain systemd-udevd[744]: worker [774] failed while handling '/devices/system/cpu/cpu1' Aug 20 21:22:03 localhost.localdomain systemd-udevd[744]: worker [777] terminated by signal 9 (KILL) Aug 20 21:22:03 localhost.localdomain systemd-udevd[744]: worker [777] failed while handling '/devices/system/cpu/cpu0' [...] I have a workaround to reduce the startup delay. Edit the udev-settle service using the command below and add the following contents to the file and reboot. The delay will be reduced from it's default of 2m to 6 seconds. Use at your own risk... I'm not sure what other side effects this will have. sudo systemctl edit systemd-udev-settle.service [Service] TimeoutSec=6 Also, sleep/suspend mode seems to fail now after this bios update. It says the below error in dmesg. Perhaps it's related to this?
Since I didn't mention this before, I'm running Fedora 28 on kernel 4.17.14.
> dmesg
[ 1827.780499] Freezing user space processes ...
[ 1847.787030] Freezing of tasks failed after 20.007 seconds (1 tasks refusing to freeze, wq_busy=0):
[ 1847.787156] systemd-udevd D 0 784 745 0x800001a4
[ 1847.787157] Call Trace:
[ 1847.787163] ? __schedule+0x234/0x840
[ 1847.787165] schedule+0x28/0x80
[ 1847.787169] __sev_do_cmd_locked+0x212/0x280 [ccp]
[ 1847.787172] ? finish_wait+0x80/0x80
[ 1847.787173] ? 0xffffffffc0530000
[ 1847.787176] __sev_platform_init_locked+0x2f/0x80 [ccp]
[ 1847.787177] ? mutex_lock+0xe/0x30
[ 1847.787179] sev_platform_init+0x1d/0x30 [ccp]
[ 1847.787182] psp_pci_init+0x40/0xe0 [ccp]
[ 1847.787183] ? 0xffffffffc0530000
[ 1847.787185] sp_mod_init+0x16/0x1000 [ccp]
[ 1847.787187] do_one_initcall+0x46/0x1c3
[ 1847.787189] ? free_unref_page_commit+0x9b/0x110
[ 1847.787190] ? _cond_resched+0x15/0x30
[ 1847.787192] ? kmem_cache_alloc_trace+0x166/0x1d0
[ 1847.787194] ? do_init_module+0x22/0x210
[ 1847.787195] do_init_module+0x5a/0x210
[ 1847.787197] load_module+0x210f/0x24a0
[ 1847.787199] ? alloc_vmap_area+0x75/0x370
[ 1847.787202] ? __do_sys_init_module+0x13f/0x180
[ 1847.787203] ? _cond_resched+0x15/0x30
[ 1847.787204] __do_sys_init_module+0x13f/0x180
[ 1847.787206] do_syscall_64+0x5b/0x160
[ 1847.787208] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 1847.787210] RIP: 0033:0x7f8c6c857ada
[ 1847.787210] RSP: 002b:00007ffd0fe933a8 EFLAGS: 00000246 ORIG_RAX: 00000000000000af
[ 1847.787212] RAX: ffffffffffffffda RBX: 00005646043a5210 RCX: 00007f8c6c857ada
[ 1847.787212] RDX: 00007f8c6d3be4cd RSI: 000000000000bbeb RDI: 00005646044b9bb0
[ 1847.787213] RBP: 00007f8c6d3be4cd R08: 0000000000000007 R09: 0000000000000006
[ 1847.787214] R10: 000056460432a010 R11: 0000000000000246 R12: 00005646044b9bb0
[ 1847.787214] R13: 000056460438c4b0 R14: 0000000000020000 R15: 0000000000000000
[ 1847.787281] OOM killer enabled.
[ 1847.787281] Restarting tasks ... done.
[ 1847.789135] thermal thermal_zone0: failed to read out thermal zone (-61)
[ 1847.789159] PM: suspend exit
The correct fix right now would be to downgrade your BIOS to an earlier version. Patch has been submitted, but not yet approved to add a timeout to a number of functions. https://marc.info/?l=linux-crypto-vger&m=153436754612783&w=2 The BIOS fix is still the correct one - however this will take some time to flush through to OEMs, and finally out to users. I can confirm the bug with MSI X470 Gaming Plus with cpu Ryzen 7 2700X. Installed fresh copy of Fedora 28 and updated all packages. Worked as expected. After upgrading BIOS to version 7B79vA4 bug happened on each boot/shutdown. After searching 2 days looking log files and googling around, i found this bug report. So i downgraded to BIOS version 7B79vA3. Boot and shutdown process went back to normal. I tried also different kernel versions with no luck. At moment kernel version is 4.17.18-200. It seems like there's no activity on the patch linked in #22 . I'm using one of the new B450 boards, the Asrock B450M Pro4 mainboard which shows the same problem. Unfortunately I can't downgrade the bios as there seems to be no bios release without the bug. Is there any other known workaround which allows using kvm until vendors decide to fix their bioses? I should suggest pinging the list / whatever area and reminding them that the patch exists and trying to push it along. The more voices, the more likely it is to happen. The patch was merged in linux-next, so it will be in 4.20/5.0 (In reply to Steven Haigh from comment #17) > Good news, I have received a reply from AMD regarding the issue after > submitting further information as gathered above and from other sources. Any update from AMD, particularly on which AGESA version might get the fix? Hopefully this doesn't get differed to the OEM as they can sometimes be unresponsive to Linux related issues. Good news on the merge. There was mention of backports after the initial merge - however time will tell if / when that occurs. No mention of a version of fixed AGESA - however it'll probably take time for various vendors to update and release a newer BIOS... *** Bug 1612608 has been marked as a duplicate of this bug. *** Since board isn't mentioned, it also applies to Gigabyte AX370 Gaming 5 + Ryzen 1600X, updated BIOS to version F23 with changelog "Update AGESA 1.0.0.4" and my system wouldn't boot. I believe I wanted more than two minutes, perhaps it would have started up after five, no idea. Also tried a live USB stick and it would not boot. Solution was to downgrade the BIOS to version F23f from May which Gigabyte, in their infinite wisdom, removed from their website when they published F23 (I luckily had a copy). F22 has some other issues so I assume they did this to ensure that there's no good BIOS for this board available on their website. As for the proposed solution of patching the kernel: That's asking the kernel to work around a software problem (advertising SEV when there isn't any) at a higher level instead of fixing the actual problem. The same applies to adding a TimeoutSec= to systemd-udev-settle.service. AMD should fix their firmware. I have an AMD Ryzen 7 1700 on an AM4 MSI X370 board and use Fedora 27 as a workaround. This problem began with me a week ago and its not limited to one distro either. Opensuse tumbleweed with kernel 4.18.5 had the system-udevd bug on shutdown, taking almost 10 minutes to fully shutdown. I have tried Manjaro as well and here is what I found with that distro. When using 4.15 kernel, I had no issues, but when I tried it with 4.18.6 the problem came back. I just tried Antergos last night and it has the same problem with 4.18.6. All the distro's were using Gnome 3.28. My system is a Ryzen 1800x, not over clocked, 32gig of ram and a Asrock Taichi x370 with bios 4.80 installed. I know this post is not releated to redhat products, but I thought letting you all know its not just redhat that's having these issues. Hope this helps. Rolled back the bios to 4.70 and now all is well. (In reply to lmmason25 from comment #32) > This problem began with me a week ago and its not limited to one distro > either. Opensuse tumbleweed with kernel 4.18.5 had the system-udevd bug on > shutdown, taking almost 10 minutes to fully shutdown. I have tried Manjaro > as well and here is what I found with that distro. When using 4.15 kernel, > I had no issues, but when I tried it with 4.18.6 the problem came back. I > just tried Antergos last night and it has the same problem with 4.18.6. All > the distro's were using Gnome 3.28. My system is a Ryzen 1800x, not over > clocked, 32gig of ram and a Asrock Taichi x370 with bios 4.80 installed. I > know this post is not releated to redhat products, but I thought letting you > all know its not just redhat that's having these issues. Hope this helps. FYI From: <ng0177> Date: Sat, Sep 15, 2018 at 10:57 AM Subject: AM4 BIOS AGESA Code 1.0.0.4C & Linux Kernel To: <brijesh.singh>, <linux-crypto.org> Hi, please find below a list of related bugs to a blocker that affects existing systems and even prevents the installation of several Linux distributions: https://bugzilla.kernel.org/show_bug.cgi?id=199267 https://bugzilla.kernel.org/show_bug.cgi?id=201129 https://bugzilla.redhat.com/show_bug.cgi?id=1608242 I read about an upstream fix to 4.20 but this is a long way to go. I guess that anyone who keeps their AM4 BIOS AGESA Code update-to-date and would like to run Linux is affected. A co-operation between AMD & Linux Kernel experts to achieve a solution fast is encouraged. Thank you! From: Brijesh Singh <brijesh.singh> Date: Sat, Sep 15, 2018 at 1:44 PM Subject: Re: AM4 BIOS AGESA Code 1.0.0.4C & Linux Kernel To: <ng0177>, <linux-crypto.org> Cc: <brijesh.singh> Hi, The workaround to handle this FW bug has been submitted last month https://marc.info/?l=linux-crypto-vger&m=153436754612783&w=2 And patch is accepted in crypto tree https://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6.git/commit/?id=3702a0585e64d70d5bf73bf3e943b8d6005b72c1 It will soon show up in Linus tree. After that we will work to get it backport to stable tree's. -Brijesh I just want to say thank you to everyone who has helped get this fixed or "fixed" depending on how you look at it and to get it backported. I have an MSI B350 Tomahawk mainboard + Ryzen 1500X + Fedora 28. I just updated today to BIOS version 7A34v1I with AGESA Code 1.0.0.4C which is the root cause for the issues as already stated in this thread. I downgraded back to version 7A34v1H with AGESA code 1.0.0.2a. After that the system booted up normally again. Heads up, the fix in noted as "crypto: ccp - add timeout support in the SEV command" has been released in Linux 4.19-rc5 as seen here: https://lwn.net/Articles/766425/ I have no idea when this will get backported, but I assume we won't see a backport until 4.19 is officially released (which should come after this RC5 afaik). Sounds good! I have to wait to install Fedora 28 or 29 until then? This is the official answer from Gigabyte: "Linux is not test be us, install OS Windows 10, work it, then is all OK. Kind regards GIGABYTE-Team Germany" (In reply to Niccolò Belli from comment #40) > This is the official answer from Gigabyte: > "Linux is not test be us, install OS Windows 10, work it, then is all OK. > > Kind regards > GIGABYTE-Team Germany" Time to find another motherboard, then. Know if any of the Threadripper-compatible companies that do test for Linux? I was sad to see that reply from Gigabyte, as I just built a system that I bought solely to use for Fedora as a virtualization workstation and for Video work. I got the Gigabyte Designare x399, which was excellent in every other way for me. I submitted my own support ticket. In the meantime, is there anyone who can verify if perhaps creating a modified version of the F28 Live .iso that has a custom kernel set to CONFIG_CRYPTO_DEV_SP_PSP=n would be a workable option? It may be a bit of work, but if I can build the kernel on another PC, and then somehow insert it in place of the one that the current live DVD iso has on it, maybe I can get this installed and running. Too bad it can't apparently be disabled by a simple kernel boot switch. (In reply to Tim from comment #42) > I was sad to see that reply from Gigabyte, as I just built a system that I > bought solely to use for Fedora as a virtualization workstation and for > Video work. I got the Gigabyte Designare x399, which was excellent in every > other way for me. > I submitted my own support ticket. > > In the meantime, is there anyone who can verify if perhaps creating a > modified version of the F28 Live .iso that has a custom kernel set to > CONFIG_CRYPTO_DEV_SP_PSP=n would be a workable option? It may be a bit of > work, but if I can build the kernel on another PC, and then somehow insert > it in place of the one that the current live DVD iso has on it, maybe I can > get this installed and running. Too bad it can't apparently be disabled by > a simple kernel boot switch. I'm not using Fedora, but recompiling my kernel with CONFIG_CRYPTO_DEV_SP_PSP=n solved my problems (I'm also using an X399 Designare Ex). I went ahead and just disabled the option on all branches. I know the timeout is coming in but that's still not a great experience for firmware. If someone really wants this at a future date, we can re-evaluate. The bug will be updated when it gets picked up in a build. @Laura, Thank you! Do you know if there will be a kernel update to Fedora 28 before 4.19 is ready? I.e. will there up a stable second kernel update available for 4.18.9? ...and to further jonathan's question, could the Live image be updated to incorporate it so that new installs can work for people with this new hardware? Yes, we will get a stable update. I don't know the exact date since I haven't seen anything queued up yet upstream but I would expect it later this week. As for updating the live images, I'm not sure how they get regenerated but once we have a kernel I can ask. I am glad that we are getting on. I have been waiting to be able to install Fedora for quite a while. Thanks now to everyone involved for their help and support! Live images for development releases are generated daily by release engineering unless something is broken. The latest workstation live images can be found in Koji[0]. Unfortunately, new live images of Fedora 28 aren't built after the final release (as far as I know) so you'll need to pick up a Fedora 29 image. You can search the "livemedia-out.log" file in the build for "vmlinuz" to figure out what kernel version is included in that image. [0] https://koji.fedoraproject.org/koji/search?match=glob&type=build&terms=Fedora-Workstation* I was hopeful in that I noted that the current download shows version 1.1, which implies there was a 1.0, making 1.2 not so much of a stretch. :) It would be nice if it could be done for 28, since it's the stable release and now with a wave of new hardware there are many orphans. The catch with 29 is that many of the 3rd party packages aren't yet caught up, but your point is good. If I find out that they for sure won't do 28, I'd rather at least have 29 running. It's been a long time since I spun my own kernel, and I've never tried to create my own spin, but this issue has me on the verge of diving in, as painful as that would be. If the team itself put a little time on it, I would think it would be fairly trivial. Check Software Updates now for kernel 4.18.10. It has fixed it for me. That's so much better now :-D I did verify that it does indeed work with the 4.18.10 kernel. I'm a bit confused as to why, however. Was it the time delay fix that was implemented, or something else? I ask because I checked CONFIG_CRYPTO_DEV_SP_PSP=y is still the config of the kernel used in 4.18.10. I tried a couple things: 1) Running the 10/1 nightly build of F29beta and that did indeed run. I didn't install it though. 2) I took a drive out of another PC that was running F28 after doing dnf update on it in that PC, so that it would be running 4.18.10. I then put that drive into the new PC with the 2950x Threadripper2 CPU and it booted and ran fine too. So if SEV isn't being disabled in the kernel config, then I wonder what the fix was, as at this point, I haven't seen that either AMD, nor Gigabyte or any other manufacturers have come out with a fixed BIOS. And if the original premise was true that the BIOS was reporting SEV to the kernel, then that should still be the case today. Whatever the case, it's nice to be able to run F28 again! Please read the kernel changelog whenever you update: https://lwn.net/Articles/766748/ Linux 4.18.10 now has the backported timeout fix/patch/workaround where there's a timer to prevent SEV query from hanging indefinitely. It's mentioned in the changelog as "crypto: ccp - add timeout support in the SEV command". Ideally AMD/Gigabyte should still release a bios fix, as I believe this workaround delays boot by a small amount (I think it's 5 seconds... Steven Haigh?). From what I understand, the max timeout will be 5 seconds by default. This is only in the module loading, so if you're using systemd, it'll launch other stuff at the same time - hence you might not actually notice this timeout. Gigabyte published X399 AORUS 7 firmware update with AGESA 1.1.0.1a. Can anyone confirm or deny that this version of AGESA fixes the issue, Linux kernel changes aside? AMD naming is a bit confusing here. AGESA 1.1.0.1a is older than 1.0.0.4c which introduced issue. Basically they restart numeration for different CPU generations and since Ryzen 2000 series latest AGESA versions end with `c` but manufacturers tend to skip last character in their patch notes. I have X470 Gaming 7 with 1.0.0.4c and with SEV timeout patch there is no issue. Like Steven Haigh said the delay isn't very noticeable because it prepares other stuff in while waiting. (In reply to Mateusz Mikuła from comment #56) > AMD naming is a bit confusing here. AGESA 1.1.0.1a is older than 1.0.0.4c > which introduced issue. Basically they restart numeration for different CPU > generations and since Ryzen 2000 series latest AGESA versions end with `c` > but manufacturers tend to skip last character in their patch notes. > > I have X470 Gaming 7 with 1.0.0.4c and with SEV timeout patch there is no > issue. Like Steven Haigh said the delay isn't very noticeable because it > prepares other stuff in while waiting. This board had AGESA 1.0.0.5 [1] prior to adding Ryzen 2000 series support (firmware version F3j) and here were no issues. I only started getting issues with 1.1.0.0. [1] https://www.gigabyte.com/Motherboard/X399-AORUS-Gaming-7-rev-10#support-dl-bios (In reply to Vedran Miletić from comment #57) > (In reply to Mateusz Mikuła from comment #56) > > AMD naming is a bit confusing here. AGESA 1.1.0.1a is older than 1.0.0.4c > > which introduced issue. Basically they restart numeration for different CPU > > generations and since Ryzen 2000 series latest AGESA versions end with `c` > > but manufacturers tend to skip last character in their patch notes. > > > > I have X470 Gaming 7 with 1.0.0.4c and with SEV timeout patch there is no > > issue. Like Steven Haigh said the delay isn't very noticeable because it > > prepares other stuff in while waiting. > > This board had AGESA 1.0.0.5 [1] prior to adding Ryzen 2000 series support > (firmware version F3j) and here were no issues. I only started getting > issues with 1.1.0.0. > > [1] > https://www.gigabyte.com/Motherboard/X399-AORUS-Gaming-7-rev-10#support-dl- > bios Nice, if it wasn't confusing enough... So ASUS/MSI released AGESA 1.1.0.1 for AM4 CPUs back in February, and 1.1.0.1a seems to be new one for TR4 CPUs. By looking at ASUS and MSI forums they have got X399 AGESA 1.1.0.1a back in August but it's still affected. It shouldn't hurt to try Gigabyte's one but don't expect too much. Not only they restart numeration for new generations, they also have separate branches for AM4 and TR4 motherboards. If you look at ASRock website, for AM4 there were SummitPI-AM4 1.0.0.4a, SummitPI-AM4 1.0.0.6, SummitPI-AM4 1.0.0.6a, SummitPI-AM4 1.0.0.6b and then PinnaclePI-AM4_1.0.0.1a (with support of Ryzen 2000), PinnaclePI-AM4_1.0.0.2a, PinnaclePI-AM4_1.0.0.4c. For TR4 there were ThreadRipperPI-SPr3-1.0.0.3, ThreadRipperPI-SPr3-1.0.0.4, ThreadRipperPI-SP3r2-1.0.0.4, then ThreadRipperPI-SP3r2 1.1.0.0 with Threadripper 2000 support (it seems that they didn't restart numeration for TR4). 1.0.0.4c version that was mentioned above is PinnaclePI-AM4_1.0.0.4c. kernel-headers-4.18.12-300.fc29 kernel-4.18.12-300.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2018-f392ab8c84 kernel-headers-4.18.12-200.fc28 kernel-4.18.12-200.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-ddbaca855e kernel-headers-4.18.12-100.fc27 kernel-4.18.12-100.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-94315e9a6b kernel-4.18.12-100.fc27, kernel-headers-4.18.12-100.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-94315e9a6b kernel-4.18.12-300.fc29, kernel-headers-4.18.12-300.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-f392ab8c84 kernel-4.18.12-200.fc28, kernel-headers-4.18.12-200.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-ddbaca855e I've encountered this issue on my brand new server which has a Asus Prime X399 motherboard running a Threadripper 2950x. I had to update to the latest 0807 bios to use 2nd gen TR, so downgrading to an older verson (0407) is not an option. The F28 server install would freeze on bootup so I cannot install F28. From the comments above, it would appear that a new kernel update resolves this SEV issue, but how does one get F28 installed first in order to update the kernel? As a stop-gap measure, I successfully installed the test release of F29 server (Fedora-Server-dvd-x86_64-29-20181002.n.0.iso). Should I worry about trying to get F28 installed? Or should I wait until F29 release date where I can update my F29 test to the final release (is that even possible?) ? (In reply to Stephen So from comment #66) > I've encountered this issue on my brand new server which has a Asus Prime > X399 motherboard running a Threadripper 2950x. I had to update to the > latest 0807 bios to use 2nd gen TR, so downgrading to an older verson (0407) > is not an option. The F28 server install would freeze on bootup so I cannot > install F28. > > From the comments above, it would appear that a new kernel update resolves > this SEV issue, but how does one get F28 installed first in order to update > the kernel? > > As a stop-gap measure, I successfully installed the test release of F29 > server (Fedora-Server-dvd-x86_64-29-20181002.n.0.iso). > > Should I worry about trying to get F28 installed? Or should I wait until > F29 release date where I can update my F29 test to the final release (is > that even possible?) ? In my experience, F29 works reasonably well at this point. I would just stick with it. Otherwise, wait for a live respin: https://dl.fedoraproject.org/pub/alt/live-respins/ kernel-4.18.12-300.fc29, kernel-headers-4.18.12-300.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report. kernel-4.18.12-200.fc28, kernel-headers-4.18.12-200.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report. kernel-4.18.12-100.fc27, kernel-headers-4.18.12-100.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report. (In reply to Vedran Miletić from comment #57) > (In reply to Mateusz Mikuła from comment #56) > > AMD naming is a bit confusing here. AGESA 1.1.0.1a is older than 1.0.0.4c > > which introduced issue. Basically they restart numeration for different CPU > > generations and since Ryzen 2000 series latest AGESA versions end with `c` > > but manufacturers tend to skip last character in their patch notes. > > > > I have X470 Gaming 7 with 1.0.0.4c and with SEV timeout patch there is no > > issue. Like Steven Haigh said the delay isn't very noticeable because it > > prepares other stuff in while waiting. > > This board had AGESA 1.0.0.5 [1] prior to adding Ryzen 2000 series support > (firmware version F3j) and here were no issues. I only started getting > issues with 1.1.0.0. > > [1] > https://www.gigabyte.com/Motherboard/X399-AORUS-Gaming-7-rev-10#support-dl- > bios In case somebody finds this issue, I want to report that I finally upgraded to X399 AORUS Gaming 7 to F11e (TR4 AGESA 1.1.0.1a) and I can confirm there is no issue with udev on 4.19.2. This comment was flagged a spam, view the edit history to see the original text if required. |
Versions: $ rpm -qa | grep systemd | sort kcm_systemd-1.2.1-10.fc28.x86_64 libreport-plugin-systemd-journal-2.9.5-1.fc28.x86_64 python2-systemd-234-5.fc28.x86_64 python3-systemd-234-5.fc28.x86_64 python-systemd-doc-234-5.fc28.x86_64 rpm-plugin-systemd-inhibit-4.14.1-9.fc28.x86_64 systemd-238-9.git0e0aa59.fc28.x86_64 systemd-bootchart-233-1.fc28.x86_64 systemd-devel-238-9.git0e0aa59.fc28.x86_64 systemd-libs-238-9.git0e0aa59.fc28.i686 systemd-libs-238-9.git0e0aa59.fc28.x86_64 systemd-pam-238-9.git0e0aa59.fc28.x86_64 systemd-udev-238-9.git0e0aa59.fc28.x86_64 When booting, I get a wait of ~2 minutes for the systemd-udev-settle service. The output after a boot shows: # systemctl status systemd-udev-settle ● systemd-udev-settle.service - udev Wait for Complete Device Initialization Loaded: loaded (/usr/lib/systemd/system/systemd-udev-settle.service; static; vendor preset: disabled) Active: failed (Result: exit-code) since Wed 2018-07-25 15:12:43 AEST; 2h 35min ago Docs: man:udev(7) man:systemd-udevd.service(8) Process: 601 ExecStart=/usr/bin/udevadm settle (code=exited, status=1/FAILURE) Main PID: 601 (code=exited, status=1/FAILURE) Jul 25 15:10:42 systemd[1]: Starting udev Wait for Complete Device Initialization... Jul 25 15:12:43 systemd[1]: systemd-udev-settle.service: Main process exited, code=exited, status=1/FAILURE Jul 25 15:12:43 systemd[1]: systemd-udev-settle.service: Failed with result 'exit-code'. Jul 25 15:12:43 systemd[1]: Failed to start udev Wait for Complete Device Initialization. # systemd-analyze blame 2min 308ms systemd-udev-settle.service 12.370s dnf-makecache.service 3.433s NetworkManager-wait-online.service 586ms systemd-fsck-root.service 436ms firewalld.service This seems to be rather new behaviour: # journalctl | grep systemd-udev-settle Jul 08 21:16:11 dracut[19925]: -rw-r--r-- 1 root root 867 May 11 20:59 usr/lib/systemd/system/systemd-udev-settle.service Jul 12 20:24:21 dracut[18886]: -rw-r--r-- 1 root root 867 May 11 20:59 usr/lib/systemd/system/systemd-udev-settle.service Jul 15 20:49:54 dracut[22107]: -rw-r--r-- 1 root root 867 May 11 20:59 usr/lib/systemd/system/systemd-udev-settle.service Jul 22 04:51:43 systemd[1]: systemd-udev-settle.service: Main process exited, code=exited, status=1/FAILURE Jul 22 04:51:43 systemd[1]: systemd-udev-settle.service: Failed with result 'exit-code'. Jul 22 15:13:46 systemd[1]: systemd-udev-settle.service: Main process exited, code=exited, status=1/FAILURE Jul 22 15:13:46 systemd[1]: systemd-udev-settle.service: Failed with result 'exit-code'. Jul 22 15:23:25 systemd[1]: systemd-udev-settle.service: Main process exited, code=exited, status=1/FAILURE Jul 22 15:23:25 systemd[1]: systemd-udev-settle.service: Failed with result 'exit-code'. Jul 22 15:37:54 dracut[18498]: -rw-r--r-- 1 root root 867 Jul 18 21:28 usr/lib/systemd/system/systemd-udev-settle.service Jul 23 04:49:40 systemd[1]: systemd-udev-settle.service: Main process exited, code=exited, status=1/FAILURE Jul 23 04:49:40 systemd[1]: systemd-udev-settle.service: Failed with result 'exit-code'. Jul 23 05:03:47 systemd[1]: systemd-udev-settle.service: Main process exited, code=exited, status=1/FAILURE Jul 23 05:03:47 systemd[1]: systemd-udev-settle.service: Failed with result 'exit-code'. Jul 23 11:46:18 systemd[1]: systemd-udev-settle.service: Main process exited, code=exited, status=1/FAILURE Jul 23 11:46:18 systemd[1]: systemd-udev-settle.service: Failed with result 'exit-code'. Jul 24 05:44:15 systemd[1]: systemd-udev-settle.service: Main process exited, code=exited, status=1/FAILURE Jul 24 05:44:15 systemd[1]: systemd-udev-settle.service: Failed with result 'exit-code'. Jul 25 10:49:26 systemd[1]: systemd-udev-settle.service: Main process exited, code=exited, status=1/FAILURE Jul 25 10:49:26 systemd[1]: systemd-udev-settle.service: Failed with result 'exit-code'. Jul 25 13:58:47 systemd[1]: systemd-udev-settle.service: Main process exited, code=exited, status=1/FAILURE Jul 25 13:58:47 systemd[1]: systemd-udev-settle.service: Failed with result 'exit-code'. Jul 25 15:12:43 systemd[1]: systemd-udev-settle.service: Main process exited, code=exited, status=1/FAILURE Jul 25 15:12:43 systemd[1]: systemd-udev-settle.service: Failed with result 'exit-code'. Are there any suspect changes that I can start looking to figure out what would cause this?