This error means that the kernel could not allocate 64kb of physically continuous memory. This likely means that your machine was running out of memory at the time this happened, is that possible ?

Unless you run out of memory regularly you are unlikely to ever hit this bug again. So I believe it is best to close this bug and re-open it in case you do happen to hit this again.

If you agree please close this bug.

I just noticed that you are indeed running out of memory more often and that you have filed bug 1829942 for that, I'll mark this bug as a duplicate of that bug then.

*** This bug has been marked as a duplicate of bug 1829942 ***

(In reply to Hans de Goede from comment #2)
> I just noticed that you are indeed running out of memory more often and that
> you have filed bug 1829942 for that, I'll mark this bug as a duplicate of
> that bug then.
> 
> *** This bug has been marked as a duplicate of bug 1829942 ***

Hans, this happened on a completely different day with no OOMs. The machine has 64GB of RAM and 32 GB of swap. The proposition that it couldn't allocate 64kB continuous is not plausible, sorry.

(In reply to Arcadiy Ivanov from comment #3)
...
> Hans, this happened on a completely different day with no OOMs. The machine has 64GB of RAM and 32 GB of swap. The proposition that it couldn't allocate 64kB continuous is not plausible, sorry.

Thanks for pointing that out. However, the kernel says that it is tainted. What non-Fedora kernel modules do you have loaded?

May 02 23:15:15 ai-karellen-lap kernel: CPU: 1 PID: 1249534 Comm: kworker/1:2 Tainted: P     U     OE     5.6.8-200.fc31.x86_64 #1
                                                                              ^^^^^^^

i915 GUC and VirtualBox. None of these errors happened before 5.6.x upgrade.

$ dmesg | grep -i taint
[    2.128772] Setting dangerous option enable_guc - tainting kernel
[   17.544724] vboxdrv: loading out-of-tree module taints kernel.
[   17.544908] vboxdrv: module verification failed: signature and/or required key missing - tainting kernel

$ modinfo vboxdrv
filename:       /lib/modules/5.6.8-200.fc31.x86_64/misc/vboxdrv.ko
version:        6.1.6 r137129 (0x002d0001)
license:        GPL
description:    Oracle VM VirtualBox Support Driver
author:         Oracle Corporation
srcversion:     1B117C52DF5B4D7EFB983ED
depends:        
retpoline:      Y
name:           vboxdrv
vermagic:       5.6.8-200.fc31.x86_64 SMP mod_unload 
parm:           force_async_tsc:force the asynchronous TSC mode (int)

(In reply to Arcadiy Ivanov from comment #5)
> i915 GUC and VirtualBox. None of these errors happened before 5.6.x upgrade.
> 
> $ dmesg | grep -i taint
> [    2.128772] Setting dangerous option enable_guc - tainting kernel
> [   17.544724] vboxdrv: loading out-of-tree module taints kernel.
> [   17.544908] vboxdrv: module verification failed: signature and/or
> required key missing - tainting kernel
...

Thanks. The stack dump shows:

May 02 23:15:15 ai-karellen-lap kernel:  dell_smbios_wmi_call+0x82/0xd0 [dell_smbios]
May 02 23:15:15 ai-karellen-lap kernel:  dell_smbios_call+0x6f/0xb0 [dell_smbios]
                                         ^^^^^^^^^^^

That appears to be a Dell-specific kernel module. Could you post the output from "lsmod"?

The background context here is that there was an ACPICA change that requires a fix in the sony-laptop module.* There could be a similar issue with the Dell modules.

A simple test would be to see if the problem ever occurs with all of the Dell modules blacklisted.

The kernel source code shows some error messages that might show up in your logs:

$ grep -C4 -n '==.*ACPI_TYPE_INTEGER' drivers/platform/x86/dell*.c
drivers/platform/x86/dell-smbios-wmi.c-65-		return -EIO;
drivers/platform/x86/dell-smbios-wmi.c-66-	obj = (union acpi_object *)output.pointer;
drivers/platform/x86/dell-smbios-wmi.c-67-	if (obj->type != ACPI_TYPE_BUFFER) {
drivers/platform/x86/dell-smbios-wmi.c-68-		dev_dbg(&wdev->dev, "received type: %d\n", obj->type);
drivers/platform/x86/dell-smbios-wmi.c:69:		if (obj->type == ACPI_TYPE_INTEGER)
drivers/platform/x86/dell-smbios-wmi.c-70-			dev_dbg(&wdev->dev, "SMBIOS call failed: %llu\n",
drivers/platform/x86/dell-smbios-wmi.c-71-				obj->integer.value);
drivers/platform/x86/dell-smbios-wmi.c-72-		return -EIO;
drivers/platform/x86/dell-smbios-wmi.c-73-	}

* Bug 1830150 - [BISECTED] Fedora 31 with kernel 5.6 does not wake up from suspend

```
$ lsmod
Module                  Size  Used by
dm_crypt               53248  1
loop                   36864  2
ccm                    20480  9
rfcomm                 90112  4
xt_MASQUERADE          20480  1
nf_conntrack_netlink    49152  0
xt_addrtype            16384  2
br_netfilter           32768  0
bridge                221184  1 br_netfilter
stp                    16384  1 bridge
llc                    16384  2 bridge,stp
nf_conntrack_netbios_ns    16384  1
nf_conntrack_broadcast    16384  1 nf_conntrack_netbios_ns
xt_CT                  16384  1
ip6t_REJECT            16384  11
nf_reject_ipv6         20480  1 ip6t_REJECT
ip6t_rpfilter          16384  1
ipt_REJECT             16384  2
nf_reject_ipv4         16384  1 ipt_REJECT
xt_conntrack           16384  22
ebtable_nat            16384  1
ebtable_broute         16384  1
ip6table_nat           16384  1
ip6table_mangle        16384  1
ip6table_raw           16384  1
ip6table_security      16384  1
iptable_nat            16384  1
nf_nat                 53248  3 ip6table_nat,iptable_nat,xt_MASQUERADE
iptable_mangle         16384  1
iptable_raw            16384  1
iptable_security       16384  1
nf_conntrack          159744  7 xt_conntrack,nf_nat,nf_conntrack_netbios_ns,nf_conntrack_broadcast,nf_conntrack_netlink,xt_CT,xt_MASQUERADE
nf_defrag_ipv6         24576  1 nf_conntrack
nf_defrag_ipv4         16384  1 nf_conntrack
overlay               135168  0
ip_set                 57344  0
nfnetlink              16384  3 nf_conntrack_netlink,ip_set
ebtable_filter         16384  1
ebtables               40960  3 ebtable_nat,ebtable_filter,ebtable_broute
ip6table_filter        16384  1
ip6_tables             32768  7 ip6table_filter,ip6table_raw,ip6table_nat,ip6table_mangle,ip6table_security
iptable_filter         16384  1
vboxnetadp             28672  0
vboxnetflt             32768  0
vboxdrv               552960  2 vboxnetadp,vboxnetflt
cmac                   16384  3
bnep                   28672  2
sunrpc                495616  1
vfat                   20480  1
fat                    86016  1 vfat
uvcvideo              114688  0
btusb                  57344  0
btrtl                  24576  1 btusb
videobuf2_vmalloc      20480  1 uvcvideo
btbcm                  20480  1 btusb
videobuf2_memops       20480  1 videobuf2_vmalloc
btintel                32768  1 btusb
videobuf2_v4l2         28672  1 uvcvideo
videobuf2_common       57344  2 videobuf2_v4l2,uvcvideo
bluetooth             630784  33 btrtl,btintel,btbcm,bnep,btusb,rfcomm
videodev              270336  3 videobuf2_v4l2,uvcvideo,videobuf2_common
mc                     61440  4 videodev,videobuf2_v4l2,uvcvideo,videobuf2_common
cdc_acm                40960  0
ecdh_generic           16384  2 bluetooth
ecc                    36864  1 ecdh_generic
snd_sof_pci            24576  0
snd_sof_intel_byt      20480  1 snd_sof_pci
snd_sof_intel_ipc      20480  1 snd_sof_intel_byt
dell_rbtn              20480  0
snd_sof_intel_hda_common    81920  1 snd_sof_pci
snd_soc_hdac_hda       24576  1 snd_sof_intel_hda_common
snd_sof_xtensa_dsp     16384  2 snd_sof_intel_hda_common,snd_sof_intel_byt
snd_sof_intel_hda      20480  1 snd_sof_intel_hda_common
snd_sof               131072  4 snd_sof_pci,snd_sof_intel_hda_common,snd_sof_intel_byt,snd_sof_intel_ipc
x86_pkg_temp_thermal    20480  0
intel_powerclamp       20480  0
coretemp               20480  0
snd_soc_skl           176128  0
snd_soc_sst_ipc        20480  1 snd_soc_skl
kvm_intel             327680  0
snd_soc_sst_dsp        40960  1 snd_soc_skl
snd_hda_ext_core       36864  4 snd_sof_intel_hda_common,snd_soc_hdac_hda,snd_soc_skl,snd_sof_intel_hda
snd_soc_acpi_intel_match    40960  3 snd_sof_pci,snd_sof_intel_hda_common,snd_soc_skl
snd_soc_acpi           16384  4 snd_soc_acpi_intel_match,snd_sof_intel_hda_common,snd_sof_intel_byt,snd_soc_skl
snd_hda_codec_hdmi     73728  1
snd_soc_core          286720  4 snd_sof,snd_sof_intel_hda_common,snd_soc_hdac_hda,snd_soc_skl
kvm                   802816  1 kvm_intel
snd_hda_codec_realtek   126976  1
snd_compress           28672  1 snd_soc_core
ac97_bus               16384  1 snd_soc_core
snd_hda_codec_generic    94208  1 snd_hda_codec_realtek
irqbypass              16384  1 kvm
snd_pcm_dmaengine      16384  1 snd_soc_core
iwlmvm                450560  0
snd_hda_intel          53248  4
snd_intel_dspcfg       28672  4 snd_hda_intel,snd_sof_pci,snd_sof_intel_hda_common,snd_soc_skl
snd_hda_codec         159744  5 snd_hda_codec_generic,snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec_realtek,snd_soc_hdac_hda
dell_laptop            28672  0
iTCO_wdt               16384  0
ledtrig_audio          16384  4 snd_hda_codec_generic,snd_hda_codec_realtek,snd_sof,dell_laptop
mac80211              999424  1 iwlmvm
snd_hda_core          106496  10 snd_hda_codec_generic,snd_hda_codec_hdmi,snd_hda_intel,snd_hda_ext_core,snd_hda_codec,snd_hda_codec_realtek,snd_sof_intel_hda_common,snd_soc_hdac_hda,snd_soc_skl,snd_sof_intel_hda
crct10dif_pclmul       16384  1
iTCO_vendor_support    16384  1 iTCO_wdt
mei_hdcp               24576  0
mei_wdt                16384  0
snd_hwdep              16384  1 snd_hda_codec
crc32_pclmul           16384  0
intel_rapl_msr         20480  0
dell_wmi               20480  0
snd_seq                86016  0
ghash_clmulni_intel    16384  0
dell_smm_hwmon         24576  0
libarc4                16384  1 mac80211
intel_cstate           20480  0
snd_seq_device         16384  1 snd_seq
dell_smbios            32768  2 dell_wmi,dell_laptop
intel_uncore          147456  0
dcdbas                 20480  1 dell_smbios
iwlwifi               339968  1 iwlmvm
snd_pcm               131072  9 snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec,snd_sof,snd_sof_intel_hda_common,snd_soc_core,snd_soc_skl,snd_hda_core,snd_pcm_dmaengine
intel_rapl_perf        20480  0
pcspkr                 16384  0
dell_wmi_descriptor    20480  2 dell_wmi,dell_smbios
snd_timer              45056  2 snd_seq,snd_pcm
wmi_bmof               16384  0
intel_wmi_thunderbolt    20480  0
cfg80211              851968  3 iwlmvm,iwlwifi,mac80211
snd                   106496  20 snd_hda_codec_generic,snd_seq,snd_seq_device,snd_hda_codec_hdmi,snd_hwdep,snd_hda_intel,snd_hda_codec,snd_hda_codec_realtek,snd_timer,snd_compress,snd_soc_core,snd_pcm
i2c_i801               32768  0
soundcore              16384  1 snd
thunderbolt           237568  0
rfkill                 28672  10 bluetooth,dell_laptop,dell_rbtn,cfg80211
processor_thermal_device    24576  0
ucsi_acpi              16384  0
mei_me                 45056  2
typec_ucsi             36864  1 ucsi_acpi
intel_rapl_common      32768  2 intel_rapl_msr,processor_thermal_device
mei                   122880  5 mei_wdt,mei_hdcp,mei_me
joydev                 28672  0
idma64                 20480  0
int3403_thermal        16384  0
intel_soc_dts_iosf     20480  1 processor_thermal_device
intel_pch_thermal      16384  0
typec                  53248  1 typec_ucsi
int340x_thermal_zone    16384  2 int3403_thermal,processor_thermal_device
intel_hid              24576  0
dell_smo8800           20480  0
sparse_keymap          16384  2 intel_hid,dell_wmi
acpi_pad              184320  0
int3400_thermal        20480  0
acpi_thermal_rel       16384  1 int3400_thermal
binfmt_misc            24576  1
ip_tables              32768  5 iptable_filter,iptable_security,iptable_raw,iptable_nat,iptable_mangle
xfs                  1642496  3
libcrc32c              16384  3 nf_conntrack,nf_nat,xfs
i915                 2490368  155905
rtsx_pci_sdmmc         32768  0
mmc_core              184320  1 rtsx_pci_sdmmc
hid_multitouch         32768  0
i2c_algo_bit           16384  1 i915
cec                    61440  1 i915
drm_kms_helper        237568  1 i915
mxm_wmi                16384  0
nvme                   49152  3
drm                   598016  27 drm_kms_helper,i915
crc32c_intel           24576  2
nvme_core             110592  5 nvme
serio_raw              20480  0
rtsx_pci               90112  1 rtsx_pci_sdmmc
i2c_hid                32768  0
pinctrl_cannonlake     36864  0
video                  53248  3 dell_wmi,dell_laptop,i915
wmi                    32768  6 intel_wmi_thunderbolt,dell_wmi,wmi_bmof,dell_smbios,dell_wmi_descriptor,mxm_wmi
pinctrl_intel          32768  1 pinctrl_cannonlake
fuse                  139264  5
```

It is a Dell Precision 5540 laptop.

```
# dmidecode 3.2
Getting SMBIOS data from sysfs.
SMBIOS 3.2.0 present.
Table at 0x000E0000.

Handle 0x0000, DMI type 0, 26 bytes
BIOS Information
        Vendor: Dell Inc.
        Version: 1.6.0
        Release Date: 02/06/2020
        Address: 0xF0000
        Runtime Size: 64 kB
        ROM Size: 32 MB
        Characteristics:
                PCI is supported
                PNP is supported
                BIOS is upgradeable
                BIOS shadowing is allowed
                Boot from CD is supported
                Selectable boot is supported
                EDD is supported
                Japanese floppy for NEC 9800 1.2 MB is supported (int 13h)
                5.25"/1.2 MB floppy services are supported (int 13h)
                3.5"/720 kB floppy services are supported (int 13h)
                3.5"/2.88 MB floppy services are supported (int 13h)
                Print screen service is supported (int 5h)
                8042 keyboard services are supported (int 9h)
                Serial services are supported (int 14h)
                Printer services are supported (int 17h)
                ACPI is supported
                USB legacy is supported
                Smart battery is supported
                BIOS boot specification is supported
                Function key-initiated network boot is supported
                Targeted content distribution is supported
                UEFI is supported
        BIOS Revision: 1.6

Handle 0x0001, DMI type 1, 27 bytes
System Information
        Manufacturer: Dell Inc.
        Product Name: Precision 5540
        Version: Not Specified
        Serial Number: xxxxxx
        UUID: xxxxxxxxxxxxxxxxxxxx
        Wake-up Type: Power Switch
        SKU Number: 0906
        Family: Precision

Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
        Manufacturer: Dell Inc.
        Product Name: 0V030K
        Version: A00
```

Thanks for the lsmod output. Let's try blacklisting dell_smbios by adding this to the kernel command-line from grub2:

module_blacklist=dell_smbios

If it looks like a lot of reboots will be required, it might be more convenient to add it in /etc/default/grub and then run grub2-mkconfig.

Before doing that, we need to try to find a reproducer. The stack dump shows set_brightness_delayed being called. From the kernel source, that appears have something to do with leds. The stack dump also shows:

May 02 23:15:15 ai-karellen-lap kernel:  kbd_get_state+0x32/0xc0 [dell_laptop]
May 02 23:15:15 ai-karellen-lap kernel:  kbd_led_level_set+0x100/0x150 [dell_laptop]
                                         ^^^^^^^^^^^^^^^^^              ^^^^^^^^^^^

There isn't enough information in your report to know when that was being called, but the kernel source shows kbd_led_level_set supports "dell::kbd_backlight" in dell-laptop.c.

Further, the last line of the stack dump shows:

May 02 23:15:15 ai-karellen-lap kernel: leds dell::kbd_backlight: Setting an LED's brightness failed (-5)
                                             ^^^^^^^^^^^^^^^^^^^

So a possible reproducer would be to toggle the keyboard backlight while monitoring the log in a full-screen terminal window with:

$ journalctl --no-hostname -k -f

==
$ cat lsmod-1.txt | egrep '^dell|Module'
Module                  Size  Used by
dell_rbtn              20480  0
dell_laptop            28672  0
dell_wmi               20480  0
dell_smm_hwmon         24576  0
dell_smbios            32768  2 dell_wmi,dell_laptop
dell_wmi_descriptor    20480  2 dell_wmi,dell_smbios
dell_smo8800           20480  0

The kernel’s command-line parameters
https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html

$ git grep set_brightness_delayed
drivers/leds/led-core.c:static void set_brightness_delayed(struct work_struct *ws)
drivers/leds/led-core.c:        INIT_WORK(&led_cdev->set_brightness_work, set_brightness_delayed);

$ git grep -C3 'kbd_led_level_set,'
drivers/platform/x86/dell-laptop.c-static struct led_classdev kbd_led = {
drivers/platform/x86/dell-laptop.c-     .name           = "dell::kbd_backlight",
drivers/platform/x86/dell-laptop.c-     .flags          = LED_BRIGHT_HW_CHANGED,
drivers/platform/x86/dell-laptop.c:     .brightness_set_blocking = kbd_led_level_set,
drivers/platform/x86/dell-laptop.c-     .brightness_get = kbd_led_level_get,
drivers/platform/x86/dell-laptop.c-     .groups         = kbd_led_groups,
drivers/platform/x86/dell-laptop.c-};

> So a possible reproducer would be to toggle the keyboard backlight ...

I'm not exactly sure how to do that, but a possible place to start would be by looking for a "brightness" file here:

$ find -L /sys/class/leds/ -maxdepth 2 2>/dev/null

And then by running something like this, but for the keyboard backlight "brightness" file:

# echo 1 > /sys/class/leds/input2::capslock/brightness
# echo 0 > /sys/class/leds/input2::capslock/brightness

(In reply to Steve from comment #10)
...
> And then by running something like this, but for the keyboard backlight "brightness" file:
> 
> # echo 1 > /sys/class/leds/input2::capslock/brightness
> # echo 0 > /sys/class/leds/input2::capslock/brightness

It might be even simpler:

May 02 23:15:15 ai-karellen-lap kernel:  kbd_get_state+0x32/0xc0 [dell_laptop]
                                             ^^^

That's a *read*, so something like this might be a reproducer:

# cat /sys/class/leds/input2::capslock/brightness

(In reply to Steve from comment #9)
> Thanks for the lsmod output. Let's try blacklisting dell_smbios by adding
> this to the kernel command-line from grub2:
> 
> module_blacklist=dell_smbios
> 
> If it looks like a lot of reboots will be required, it might be more
> convenient to add it in /etc/default/grub and then run grub2-mkconfig.

I am confused as to what this is trying to accomplish by this. This error happened once and so far I haven't been able to reproduce it since.

`dell::kbd_backlight` is Dell keyboard backlight control and it's trivial to change the value by either keyboard combo or directly via /sys:

```
$ cat /sys/class/leds/dell::kbd_backlight/brightness
1
$ cat /sys/class/leds/dell::kbd_backlight/brightness
0
$ cat /sys/class/leds/dell::kbd_backlight/brightness
2
$ cat /sys/class/leds/dell::kbd_backlight/brightness
0
```

But I think we're focusing on the wrong thing here. This is the reason I filed this bug:

```
May 02 23:15:15 ai-karellen-lap kernel:  ? __alloc_pages_direct_compact+0x144/0x150
May 02 23:15:15 ai-karellen-lap kernel:  __alloc_pages_slowpath+0xd99/0xdd0
May 02 23:15:15 ai-karellen-lap kernel:  ? _cond_resched+0x15/0x30
May 02 23:15:15 ai-karellen-lap kernel:  __alloc_pages_nodemask+0x2c9/0x310
May 02 23:15:15 ai-karellen-lap kernel:  kmalloc_order+0x1b/0x80
May 02 23:15:15 ai-karellen-lap kernel:  kmalloc_order_trace+0x1d/0xa0
May 02 23:15:15 ai-karellen-lap kernel:  acpi_ut_initialize_buffer+0x3b/0x6f
May 02 23:15:15 ai-karellen-lap kernel:  acpi_evaluate_object+0x1d1/0x226
May 02 23:15:15 ai-karellen-lap kernel:  wmidev_evaluate_method+0x102/0x130 [wmi]
May 02 23:15:15 ai-karellen-lap kernel:  ? __switch_to_asm+0x40/0x70
May 02 23:15:15 ai-karellen-lap kernel:  ? __switch_to_asm+0x34/0x70
May 02 23:15:15 ai-karellen-lap kernel:  run_smbios_call+0x59/0x170 [dell_smbios]
```

`kbd_get_state` invoked `wmidev_evaluate_method`, which invoked `acpi_evaluate_object`, which in turned asked kmalloc for memory and kmalloc, as Hans noted, failed to allocate a 64kB buffer. Considering that machine kept chugging after that with no problems, i.e. the kernel wasn't actually out of memory to the point of not being able to allocate 16 pages, would suggest there may be a kmalloc bug.

Here is another bug all the way from 3.19 that leads along the similar path: https://bugzilla.redhat.com/show_bug.cgi?id=965962
Bug 965962 suggests it may be a firmware bug.