{ping}

{update}
No change in behaviour with new version:
dracut-057-3.fc36.x86_64
dracut-config-rescue-057-3.fc36.x86_64

{ping2}
Why isn't there any feedback?

The cause of the issue is that AMD microcode is compressed in linux-firmware package, and that dracut does not support compressed microcode binaries. You can workaround the issue by executing the following command inside /lib/firmware/amd-ucode:

sudo xz -d *

...and regenerating initramfs.

This likely belongs to linux-firmware, not dracut. Fedora's linux-firmware has a patch that compressed everything listed in WHENCE. Since dracut has never supported compressed microcode and WHENCE has listed AMD microcode files at least for as long as compression patch exists in Fedora, AMD probably didn't have early microcode in Fedora since February 2021 (when the patch was introduced).

(In reply to Sergey Kozharinov from comment #4)
> The cause of the issue is that AMD microcode is compressed in linux-firmware
> package, and that dracut does not support compressed microcode binaries. You
> can workaround the issue by executing the following command inside
> /lib/firmware/amd-ucode:
> 
> sudo xz -d *
> 
> ...and regenerating initramfs.

Yes, works this way.👍
At least a new workaround.

> This likely belongs to linux-firmware, not dracut. Fedora's linux-firmware
> has a patch that compressed everything listed in WHENCE. Since dracut has
> never supported compressed microcode and WHENCE has listed AMD microcode
> files at least for as long as compression patch exists in Fedora, AMD
> probably didn't have early microcode in Fedora since February 2021 (when the
> patch was introduced).

I see. Install with compression.

'make DESTDIR=%{buildroot}/ FIRMWAREDIR=%{_firmwarepath} installcompress'

Would be easy fix to just uncompress /lib/firmware/amd-ucode after the install line in the spec file. Same as removal of unwanted files and moving some files/folders.

Assigning to linux-firmware. Suggestion to make early microcode loading work again on AMD:

Uncompress /lib/firmware/amd-ucode after the install with installcompress in spec file.

See comments #4 and #5.

This is a bug in dracut, please do not move it to the wrong component.

(In reply to Peter Robinson from comment #7)
> This is a bug in dracut, please do not move it to the wrong component.

Dracut misses a feature OK, but I don't consider a missing feature (compression support) a bug.

But let's say, it's a bug in dracut. As this discussion leads nowhere.
A fix in linux-firmware really simple. Why not just do it for now, as the user experience is the real thing that matters?

Just looked at the linux-firmware-20221012-r1.ebuild from Gentoo. They compress every file in the linux-firmware *except* amd-ucode directory.

(In reply to Sergey Kozharinov from comment #9)
> Just looked at the linux-firmware-20221012-r1.ebuild from Gentoo. They
> compress every file in the linux-firmware *except* amd-ucode directory.

In archlinux amd-ucode/microcode_amd*.bin is also handled separately. They put a combined "amd-ucode.img" into a sub package.
Plenty of options. And no need to have uncompressed ucode files in the main package. If that really matters.

I'm curious, if a kernel maintainer from cc: picks this up and puts the few lines into the linux-firmware spec...

So it seems, there are currently three distributions compressing firmware (Fedora, Gentoo, Arch and derivatives).

Conclusion:
Two distributions have working AMD microcode patching and handled it in linux-firmware packaging. Both are using dracut. 
One distribution with especially one linux-firmware maintainer claims a bug in dracut and leaves AMD microcode broken without activity and/or explanation. 

Perfectly fine to fake check me. Comments welcome.

(In reply to samoht0 from comment #11)
> So it seems, there are currently three distributions compressing firmware
> (Fedora, Gentoo, Arch and derivatives).
> 
> Conclusion:
> Two distributions have working AMD microcode patching and handled it in
> linux-firmware packaging. Both are using dracut. 
> One distribution with especially one linux-firmware maintainer claims a bug
> in dracut and leaves AMD microcode broken without activity and/or
> explanation. 

There has been explanations, just because you choose to ignore them is not my problem.

The kernel handles compressed firmware just fine for literally 1000s of firmwares, if dracut can't handle is for one specific set of firmware it's a bug in dracut. From a 2 second grep it handles it here:
https://github.com/dracutdevs/dracut/blob/master/dracut.sh#L2253

If two other distros handle it "correctly", and no special casing what firmware is compressed or not is not fixing it, it's a hack to work around a bug elsewhere, why haven't you references how they do it in their packaging.

> Perfectly fine to fake check me. Comments welcome.

Patches are even more welcome. Comments are easy, fixing the problem seemingly less so. If you're so pent up with getting it fixed references to those other distros and more details rather than pointless "comments welcome" may actually get you closer to a solution to fixing your bug. Pointless unhelpful comments and "+1"s or "me too"s definitely will not.

Created attachment 1924015 [details]
linux-firmware AMD ucode patch

(In reply to Peter Robinson from comment #12)
> There has been explanations, just because you choose to ignore them is not
> my problem.
> 
> The kernel handles compressed firmware just fine for literally 1000s of
> firmwares, if dracut can't handle is for one specific set of firmware it's a
> bug in dracut. From a 2 second grep it handles it here:
> https://github.com/dracutdevs/dracut/blob/master/dracut.sh#L2253

True, but as noted above "dracut does not support compressed microcode binaries". Spicing this in at distribution level means: Amateurs are hacking around a complex script, that is vital for system boot-up. This not my department and I'd strongly recommend to never do such things. As there's no upstream patch, there's no backport, too.

> If two other distros handle it "correctly", and no special casing what
> firmware is compressed or not is not fixing it, it's a hack to work around a
> bug elsewhere, why haven't you references how they do it in their packaging.

Compressing firmware is unsupported by kernel upstream. Considering fixing AMD ucode a hack, compressing itself would be a hack even more clearly.
Arch and Gentoo have public git repos. My point was, they handle it in linux-firmware. The solution is isn't fully clear to me. Maybe it's boot config dependent. So I didn't recommend to copy it. I strongly discourage to do so.

Just for the record, this are the current package definitions:
https://github.com/archlinux/svntogit-packages/tree/packages/linux-firmware/trunk
https://gitweb.gentoo.org/repo/gentoo.git/tree/sys-kernel/linux-firmware/linux-firmware-20221109.ebuild

> Patches are even more welcome. Comments are easy, fixing the problem
> seemingly less so. If you're so pent up with getting it fixed references to
> those other distros and more details rather than pointless "comments
> welcome" may actually get you closer to a solution to fixing your bug.
> Pointless unhelpful comments and "+1"s or "me too"s definitely will not.

Ok, if this is really the matter: I've done those tiny changes and package builds fine.

(In reply to samoht0 from comment #14)
> Compressing firmware is unsupported by kernel upstream.

This references the copy/install script provided by kernel.org firmware package.

(In reply to samoht0 from comment #15)
> (In reply to samoht0 from comment #14)
> > Compressing firmware is unsupported by kernel upstream.
> 
> This references the copy/install script provided by kernel.org firmware
> package.

The kernel itself supports compressed firmware just fine

What's the state of checking/applying the proposed packaging patch?

(In reply to samoht0 from comment #17)
> What's the state of checking/applying the proposed packaging patch?

I will not be applying any patch to linux-firmware, and I've not seen any proposed fix for dracut, so there is no state

> The kernel handles compressed firmware just fine for literally 1000s of firmwares

CPU microcode isn't firmware. And Linux kernel (6.1-rc6) handles it differently:

x86_64_start_kernel(...)
-> load_ucode_bsp()
  -> load_ucode_amd_bsp(cpuid_1_eax)
    -> __load_ucode_amd(cpuid_1_eax, ret)
      -> find_microcode_in_initrd(path="kernel/x86/microcode/AuthenticAMD.bin", use_pa=false)
        -> find_cpio_data(path, start, size, NULL)
    -> apply_microcode_early_amd(cpuid_1_eax, ucode, size, save_patch=true)
      -> scan_containers(ucode, size, desc)
        -> parse_container(ucode, size, desc)

None of these functions try to do anything even remotely close to decompression.
I am also unsure if kernel can allocate memory this early, if not - decompression is probably impossible.

All of the above makes me wonder if AMD CPU microcode belogns to a subpackage.

(In reply to Peter Robinson from comment #18)
> I will not be applying any patch to linux-firmware, and I've not seen any
> proposed fix for dracut, so there is no state

This is my understanding: You're refusing to apply a patch, that is tested to fix the bug and has no packaging issues. Because dracut needs a patch, even all other distributions that use compression handle the problem in firmware packaging.
This isn't some package lottery. There's a obvious reason to do it that way. It just works.
It's not like I'm depending on your cooperation. Was a try to follow the advertised fedora principle:

"It is built and used by people across the globe who work together as a community."

Well, mostly it's like that. But as shown here: Sometimes totally the opposite, because of ... (some reason here).

I'll put the uncompressed files into a separate package. Works for me. Fedora stays broken. Perfectly fine by me.

(In reply to Sergey Kozharinov from comment #19)
> > The kernel handles compressed firmware just fine for literally 1000s of firmwares
> 
> CPU microcode isn't firmware. And Linux kernel (6.1-rc6) handles it
> differently:
> 
> x86_64_start_kernel(...)
> -> load_ucode_bsp()
>   -> load_ucode_amd_bsp(cpuid_1_eax)
>     -> __load_ucode_amd(cpuid_1_eax, ret)
>       ->
> find_microcode_in_initrd(path="kernel/x86/microcode/AuthenticAMD.bin",
> use_pa=false)
>         -> find_cpio_data(path, start, size, NULL)
>     -> apply_microcode_early_amd(cpuid_1_eax, ucode, size, save_patch=true)
>       -> scan_containers(ucode, size, desc)
>         -> parse_container(ucode, size, desc)
> 
> None of these functions try to do anything even remotely close to
> decompression.
> I am also unsure if kernel can allocate memory this early, if not -
> decompression is probably impossible.
> 
> All of the above makes me wonder if AMD CPU microcode belogns to a
> subpackage.

You've got a good point. Arch uses the package "amd-ucode". But I wanted the changes to be as minor as possible, to make changes nicely reviewable and acceptable.

As I need to fix it myself, I'll use an extra package alongside. A bit wonky, but better than file management by hand ("We're not on Windows.").

Thanks for your helpful hints and comments.

Hm, am I understanding this correctly?

 • Upstream firmware is not compressed.
 • dracut does not handle compressed firmware.
 • In Fedora, we made a change to the linux-firmware package to compress the files on disk.
 • AMD early microcode loading no longer works until/unless dracut learns to handle compression.

Typically, we try to make sure that we don't just make changes in one package, that depend on features that haven't yet been implemented in another package and then "break" things until that new feature has been implemented.

That's kind of the point of a Linux distribution — to curate versions and configurations of each downstream software component, that work together correctly. 

I understand that the best answer is to fix dracut to handle the compression, but that's not the point. From the point of view of the distribution overall, this is wrong. We expect package maintainers to care about the whole distribution working together properly, not just consider their own packages in isolation.

I feel very strongly that the right thing for the distribution is to revert the compression (at least for the AMD microcode) until dracut can cope.

I'm going to reopen this ticket; if you feel equally strongly that you want it closed, let's consult FESCo for adjudication? That's the kind of thing they're there for.

(In reply to David Woodhouse from comment #21)
> Hm, am I understanding this correctly?
> 
>  • Upstream firmware is not compressed.
>  • dracut does not handle compressed firmware.
>  • In Fedora, we made a change to the linux-firmware package to compress the
> files on disk.
>  • AMD early microcode loading no longer works until/unless dracut learns to
> handle compression.
> 
> Typically, we try to make sure that we don't just make changes in one
> package, that depend on features that haven't yet been implemented in
> another package and then "break" things until that new feature has been
> implemented.
> 
> That's kind of the point of a Linux distribution — to curate versions and
> configurations of each downstream software component, that work together
> correctly. 
> 
> I understand that the best answer is to fix dracut to handle the
> compression, but that's not the point. From the point of view of the
> distribution overall, this is wrong. We expect package maintainers to care
> about the whole distribution working together properly, not just consider
> their own packages in isolation.
> 
> I feel very strongly that the right thing for the distribution is to revert
> the compression (at least for the AMD microcode) until dracut can cope.
> 
> I'm going to reopen this ticket; if you feel equally strongly that you want
> it closed, let's consult FESCo for adjudication? That's the kind of thing
> they're there for.

Dracut can handle compressed firmware. But microcode is handled differently both in dracut and in kernel. If dracut were to support compressed microcode, it would have to decompress it before combining it with initrd. Seems like a waste of computational resources with no real benefits since packages are usually compressed regardless.

(In reply to David Woodhouse from comment #21)
> Hm, am I understanding this correctly?
> 
>  • Upstream firmware is not compressed.
>  • dracut does not handle compressed firmware.
>  • In Fedora, we made a change to the linux-firmware package to compress the
> files on disk.
>  • AMD early microcode loading no longer works until/unless dracut learns to
> handle compression.

You're right, David. That are the key points. Sergey already noted, that CPU microcode is a speciality within the upstream kernel firmware files.
And we're speaking of tiny files. Compressing AMD microcode .bin files shrinks them to 31,6 KiB coming from 49,4 KiB.

This message is a reminder that Fedora Linux 36 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 36 on 2023-05-16.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '36'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version. Note that the version field may be hidden.
Click the "Show advanced fields" button if you do not see it.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 36 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

FEDORA-2023-31e2ef80b9 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-31e2ef80b9

FEDORA-2023-eeff822011 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2023-eeff822011

FEDORA-2023-eeff822011 has been pushed to the Fedora 37 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-eeff822011`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-eeff822011

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2023-31e2ef80b9 has been pushed to the Fedora 38 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-31e2ef80b9`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-31e2ef80b9

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2023-31e2ef80b9 has been pushed to the Fedora 38 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-31e2ef80b9`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-31e2ef80b9

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2023-eeff822011 has been pushed to the Fedora 37 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-eeff822011`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-eeff822011

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2023-31e2ef80b9 has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2023-eeff822011 has been pushed to the Fedora 37 stable repository.
If problem still persists, please make note of it in this bug report.