Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 2164594
| Summary: | Systems that update from systemd-252.4-4.fc38 to systemd-253~rc1-1.fc38 fail to boot | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Adam Williamson <awilliam> | ||||||
| Component: | systemd | Assignee: | systemd-maint | ||||||
| Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||
| Severity: | urgent | Docs Contact: | |||||||
| Priority: | unspecified | ||||||||
| Version: | rawhide | CC: | darrellpf, dtardon, fedoraproject, filbranden, lnykryn, msekleta, robatino, ryncsn, systemd-maint, yuwatana, zbyszek | ||||||
| Target Milestone: | --- | ||||||||
| Target Release: | --- | ||||||||
| Hardware: | All | ||||||||
| OS: | Linux | ||||||||
| Whiteboard: | openqa | ||||||||
| Fixed In Version: | systemd-253~rc1-3.fc38 | Doc Type: | If docs needed, set a value | ||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2023-01-26 10:51:09 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Bug Depends On: | |||||||||
| Bug Blocks: | 2083910 | ||||||||
| Attachments: |
|
||||||||
|
Description
Adam Williamson
2023-01-25 18:53:04 UTC
With logs redirected somewhere I can see 'em but not at debug level, I see 'systemd-journald.service: Scheduled restart job, restart counter is at 1.', then repeating 'Looping too fast. Throttling execution a little.' messages. In rescue mode it gets stuck on fsck Adding no fsck parameter gets to Mounting sysroot mount - /sysroot ... Then hangs Created attachment 1940497 [details]
debug boot log
OK, here's a full debug-level boot log up to the point where the 'systemd-journald-audit.socket' messages start looping.
I think initrd needs to be regenerated, otherwise the socket is still started from there. (In reply to David Tardon from comment #4) > I think initrd needs to be regenerated, otherwise the socket is still > started from there. No, scratch that. That should workaround the problem, but it's not a fix. The socket should be enabled via presets after the update, but it looks like it isn't? Hmm, so I'm testing with a VM here, and it fails reliably with kernel-6.2.0-0.rc4.20230120gitd368967cb103.35.fc38.x86_64. But with kernel-6.1.0-0.rc2.21.fc38.x86_64, things work fine. With the 6.2 kernel, I'm getting an OOPS with a warning, triggered by udevd, about some mapping being done wrong (I'll try to capture it properly later). And very strange messages from udev, that stink of memory corruption. /dev/vda is not detected at all by the kernel. I'll try to figure out what is going on tomorrow. It's close to midnight and I need catch a nap. Created attachment 1940523 [details]
unexpected message from udev-builtin-keyboard
Booting a 6 1 kernel didn't work for me. I tried downgrading 1) boot a live usb 2) download previous systemd 3) mount the old root via gnome disks and chroot to it 4) verify dnf said it saw the new version 5) rpm -Uvh --oldpackage the older systemd version When I reboot the bad RC version is still there. I'm sure I've done this in the past. What step did I miss? as long as you mounted the right partition and chroot'ed properly, that should be right...maybe you missed a step, just try again? I tend to use dnf downgrade rather than rpm, but it shouldn't matter. oh, and you'll want to do all subpackages of systemd and downgrade them all together; I usually use `koji download-build --arch=x86_64 --arch=noarch systemd-252.4-4.fc38` (or whatever package and arch), then `dnf downgrade *.rpm`. FEDORA-2023-326cfb9cf8 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-326cfb9cf8 FEDORA-2023-326cfb9cf8 has been pushed to the Fedora 38 stable repository. If problem still persists, please make note of it in this bug report. https://github.com/systemd/systemd/issues/26216 is an upstream bug about PID1 handling this badly. |