Bug 1083716

Well spotted.  CC'ing Harald.

OK, so if we wanted (needed?) this we'd have to make sure that kernel-install called dracut appropriately.  At the least the kernel-install/dracut changes should land first.

From a practical point of view, do we really need early microcode loading?  What benefit does it get us other than "it's loaded earlier"?

A better option for dracut is to update the
  /usr/lib/dracut/dracut.conf.d/01-dist.conf
file and set
  early_microcode="yes"


The benefit is a shorter window where code is subject to problems due to a hardware errata.  On systems where I've tested this, the dmesg logs show that the microcode is updated almost immediately on boot with CPU0 happening right at 0.0 seconds:

[    0.000000] CPU0 microcode updated early to revision 0x29, date = 2013-06-12
[    0.048193] CPU1 microcode updated early to revision 0x29, date = 2013-06-12
[    0.061528] CPU2 microcode updated early to revision 0x29, date = 2013-06-12
[    0.074800] CPU3 microcode updated early to revision 0x29, date = 2013-06-12


Compare to the current method which updates when the microcode.ko module is loaded which happened at about 7 seconds after boot on this same system:

[    6.912947] microcode: CPU0 sig=0x206a7, pf=0x2, revision=0x28
[    7.089009] microcode: CPU0 sig=0x206a7, pf=0x2, revision=0x28
[    7.089468] microcode: CPU0 updated to revision 0x29, date = 2013-06-12


If this benefit isn't worth it, then the config-x86-generic file should remove the setting CONFIG_MICROCODE_INTEL_EARLY=y since that's a bit misleading.

Oh, and if the initramfs image is missing the microcode data, it is possible to update the microcode later in boot by running
    echo 1 > /sys/devices/system/cpu/microcode/reload

A systemd one-shot service can run this as a failsafe later in boot, for example:

=========================================
/usr/lib/systemd/system/microcode.service
=========================================
[Unit]
Description=Load CPU microcode update
After=basic.target
ConditionVirtualization=false

[Service]
Type=oneshot
RemainAfterExit=no
ExecStart=/usr/bin/bash -c "echo 1 > /sys/devices/system/cpu/microcode/reload"

[Install]
WantedBy=basic.target

(In reply to Jeff Bastian from comment #2)
> A better option for dracut is to update the
>   /usr/lib/dracut/dracut.conf.d/01-dist.conf
> file and set
>   early_microcode="yes"

Sure.  Still requires a dracut change.

> The benefit is a shorter window where code is subject to problems due to a
> hardware errata.  On systems where I've tested this, the dmesg logs show

Which, in reality, has happened how often?  I honestly don't know, but I can't imagine it is very often or we'd have heard of it long before now.

> that the microcode is updated almost immediately on boot with CPU0 happening
> right at 0.0 seconds:
> 
> [    0.000000] CPU0 microcode updated early to revision 0x29, date =
> 2013-06-12
> [    0.048193] CPU1 microcode updated early to revision 0x29, date =
> 2013-06-12
> [    0.061528] CPU2 microcode updated early to revision 0x29, date =
> 2013-06-12
> [    0.074800] CPU3 microcode updated early to revision 0x29, date =
> 2013-06-12
> 
> 
> Compare to the current method which updates when the microcode.ko module is
> loaded which happened at about 7 seconds after boot on this same system:
> 
> [    6.912947] microcode: CPU0 sig=0x206a7, pf=0x2, revision=0x28
> [    7.089009] microcode: CPU0 sig=0x206a7, pf=0x2, revision=0x28
> [    7.089468] microcode: CPU0 updated to revision 0x29, date = 2013-06-12

Right, that just shows the config option does what it says it does.

> If this benefit isn't worth it, then the config-x86-generic file should
> remove the setting CONFIG_MICROCODE_INTEL_EARLY=y since that's a bit
> misleading.

I'm not saying it's not worthwhile.  I'm just curious how much it will actually change.  It seems to be a somewhat precautionary thing to do and I'm concerned about changing this at this point.  Also, AMD has an early option as well.  I need to figure out what happens when we build both Intel and AMD early microcode support into the kernel.

I do agree we should drop the current config setting if we don't build MICROCODE=y

(In reply to Jeff Bastian from comment #3)
> Oh, and if the initramfs image is missing the microcode data, it is possible
> to update the microcode later in boot by running
>     echo 1 > /sys/devices/system/cpu/microcode/reload
> 
> A systemd one-shot service can run this as a failsafe later in boot, for
> example:
> 
> =========================================
> /usr/lib/systemd/system/microcode.service
> =========================================
> [Unit]
> Description=Load CPU microcode update
> After=basic.target
> ConditionVirtualization=false
> 
> [Service]
> Type=oneshot
> RemainAfterExit=no
> ExecStart=/usr/bin/bash -c "echo 1 >
> /sys/devices/system/cpu/microcode/reload"
> 
> [Install]
> WantedBy=basic.target

Ugh...  is that _required_ to get any microcode loaded if it isn't in the initramfs, or does the kernel try early, then automatically try later if it didn't find anything early?  I'll try looking tomorrow but if you happen to know, I'd appreciate it.

I would rather dislike having to ship a systemd service just to get microcode loaded when it's currently loaded without it now.  Particularly since that one-shot isn't needed if the microcode _is_ loaded early.

As best I can tell, the service is required, unfortunately.

In the current setup, the microcode update happens when the microcode.ko module is loaded.  If the microcode driver is built-in, the kernel checks when the system boots which means it's needed in the initramfs image, and if it's not there it does not get updated.

Personally, I don't care either way -- built-in or module -- I just noticed the conflict in the config settings.

Seems this was all done in RHEL7 already.  Why it wasn't done in Fedora first, I have no idea....

So in RHEL7 it seems this requires changes in 3 components:

1) The kernel configs and Requires
2) dracut to both create and understand (lsinitrd) initramfses with microcode attached
3) microcode_ctl to install the service mentioned and toggle the default for dracut to on for microcode.

Item 1 is "easy" to do from a config standpoint.  We need to adjust the Requires on dracut to >= 033 I think, which would cover item 2.  Harald is that correct?  If so, this won't work on F19 as dracut is too old.  F20 and rawhide would be OK.

Item 3 requires Anton (now CC'd) to update microcode_ctl and add the conf and service files.

Thinking about this more, it would seem better to me to have the kernel microcode loader do one last automatic check if/when the kernel pivots out of the initramfs to the real root.  That would eliminate the need for the service file.  I'll see if I can bug upstream about that.

dracut in RHEL-7 and rawhide is ready.
F20 with https://admin.fedoraproject.org/updates/FEDORA-2014-4704/dracut-037-10.git20140402.fc20

OK, with the haswell TSX issue, we clearly have a need for this.  I should have done this a long time ago, so my apologies.

I've enabled the kernel options today, and confirmed that rebuilding the initramfs by passing --early-microcode works.

[jwboyer@nuc-i7 ~]$ dmesg | grep micro
[    0.000000] CPU0 microcode updated early to revision 0x18, date = 2014-05-23
[    0.145247] CPU1 microcode updated early to revision 0x18, date = 2014-05-23
[    0.952104] microcode: CPU0 sig=0x40651, pf=0x40, revision=0x18
[    0.952127] microcode: CPU1 sig=0x40651, pf=0x40, revision=0x18
[    0.952152] microcode: CPU2 sig=0x40651, pf=0x40, revision=0x18
[    0.952179] microcode: CPU3 sig=0x40651, pf=0x40, revision=0x18
[    0.952414] microcode: Microcode Update Driver: v2.00 <tigran.co.uk>, Peter Oruba
[jwboyer@nuc-i7 ~]$ 


So now the question is, how do we get this option enabled by default in dracut and/or how do we pass it via kernel-install?

We'll likely also need the one-shot service, but that can probably wait.

Created attachment 941217 [details]
Patch to enable early-microcode in dracut by default

Thinking about it more, if we're always going to enable it on kernel install then it might as well be the default in dracut as well.  The attached patch switches this on by default in dracut.

I tested a local build of F21 kernel and dracut with this change and it performed as expected.  To make it easier for others to test I started a COPR that included these packages.  You can find it here (once the builds complete):

https://copr.fedoraproject.org/coprs/jwboyer/early-microcode/

dracut-038-29.git20140903.fc21,kernel-3.16.3-302.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/dracut-038-29.git20140903.fc21,kernel-3.16.3-302.fc21

I don't have a Haswell system handy, but I using your copr packages I did verify that the microcode loaded early on my Lenovo T60:

[jmbastia@firefly ~]$ uname -r
3.16.3-301.fc21.x86_64

[jmbastia@firefly ~]$ rpm -q dracut
dracut-038-29.git20140903.fc21.x86_64

[jmbastia@firefly ~]$ dmesg | grep -i microcode
[    0.000000] CPU0 microcode updated early to revision 0xd1, date = 2010-10-01
[    0.002000] CPU1 microcode updated early to revision 0xd1, date = 2010-10-01
[    0.499953] microcode: CPU0 sig=0x6f6, pf=0x20, revision=0xd1
[    0.499964] microcode: CPU1 sig=0x6f6, pf=0x20, revision=0xd1
[    0.500064] microcode: Microcode Update Driver: v2.00 <tigran.co.uk>, Peter Oruba

[jmbastia@firefly ~]$ sudo cat /sys/devices/system/cpu/cpu0/microcode/version
0xd1

Package dracut-038-29.git20140903.fc21, kernel-3.16.3-302.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing dracut-038-29.git20140903.fc21 kernel-3.16.3-302.fc21'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-11661/dracut-038-29.git20140903.fc21,kernel-3.16.3-302.fc21
then log in and leave karma (feedback).

dracut-038-29.git20140903.fc21, kernel-3.16.3-302.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.