Created attachment 1135700 [details]
OpenSSL Speed Test on 4.4.4-301

Created attachment 1135705 [details]
SysBench Test on 4.2.3-300

Created attachment 1135706 [details]
SysBench Test on 4.4.4-301

I just tested against all prior versions of kernel released through "updates", and everything up through version 4.4.3-300 worked normally, while version 4.4.4-300 is where the slow down occurs.  I have also tested against version 4.4.5-301 currently in "updates-testing" repository, and the problem persists.

This seems to have to do with the Intel SpeedStep governors.  In 4.4.3 and prior, my 2.40 MHz processor would fluctuate between 1000 and 3400 MHz.  In 4.4.4, the processor would fluctuate between 400 and 700 MHz, according to /proc/cpuinfo.

Setting /sys/devices/system/cpu/cpufreq/policy0/scaling_governor to performance, instead of the default "powersave" forces the CPU to 2400 MHz, and improves performance greatly, but still not to the same level as in 4.4.3.

Attached /proc/cpuinfo from both kernels, while under load.

Created attachment 1136112 [details]
/proc/cpuinfo for kernel 4.4.3-300

Created attachment 1136113 [details]
/proc/cpuinfo for kernel 4.4.4-301

(In reply to Jason H. from comment #5)
> This seems to have to do with the Intel SpeedStep governors.  In 4.4.3 and
> prior, my 2.40 MHz processor would fluctuate between 1000 and 3400 MHz.  In
> 4.4.4, the processor would fluctuate between 400 and 700 MHz, according to
> /proc/cpuinfo.
> 
> Setting /sys/devices/system/cpu/cpufreq/policy0/scaling_governor to
> performance, instead of the default "powersave" forces the CPU to 2400 MHz,
> and improves performance greatly, but still not to the same level as in
> 4.4.3.
> 
> Attached /proc/cpuinfo from both kernels, while under load.

This is also affecting me and has ever since I installed the updates for this kernel.  The above helped me get to usable performance but it is still degraded.

I'm not seeing any degradation on sysbench when I run on my machine with 4.4.5. I'm also not seeing any major changes to SpeedStep between 4.4.3 and 4.4.4 or any other system that jumps out at me. Can you share the following

1) dmesg from working and non-working kernels
2) lspci

You can also try the bisection scripts at https://pagure.io/fedbisect to see which commit between 4.4.3 and 4.4.4 may have broken it.

Created attachment 1136316 [details]
lspci (Jon W.)

Created attachment 1136317 [details]
dmesg 4.4.4-301.fc23.x86_64

At this point I can only get what is currently running at 4.4.4 as I am still using the computer.  I will have get the old dmesg later.

Created attachment 1136322 [details]
dmesg 4.2.3-300

Created attachment 1136323 [details]
dmesg 4.4.4-301

Created attachment 1136324 [details]
lspci

The Bisect scripts failed miserably:
$ ./fedbisect.sh sync something
~/fedbisect/scripts ~/fedbisect
Traceback (most recent call last):
  File "./fedbisect-run.py", line 3, in <module>
    import bisect_state
  File "/home/jason/fedbisect/scripts/bisect_state.py", line 5, in <module>
    import koji_cli
  File "/home/jason/fedbisect/scripts/koji_cli.py", line 29, in <module>
    import koji
ImportError: No module named koji


After installing "koji" package:

$ ./fedbisect.sh sync something
~/fedbisect/scripts ~/fedbisect
Traceback (most recent call last):
  File "./fedbisect-run.py", line 3, in <module>
    import bisect_state
  File "/home/jason/fedbisect/scripts/bisect_state.py", line 8, in <module>
    from git import Repo
ImportError: No module named git


I installed every package for python that includes the name "git", and couldn't get it working.

Please include instructions on all required prerequisites.

Sorry, the docs are missing the dependencies. I'm going to update them. The
packages you should need are

   koji
   GitPython
   fedpkg
   hmaccalc
   pesign
   gcc

Created attachment 1136645 [details]
dmesg 4.4.3-300.fc23.x86_64

I ran through fedbisect, it returned "Found your commit!".  I did a little research about git bisect to get the details.  I would recommend adding a section in your script to output the last bad commit details when it is found, or add to your readme what to do when "Found your commit" is displayed.


"git bisect log" returns:

# first bad commit: [774ac8b7eff69e0786970157de2157e68b22f456] Thermal: initialize thermal zone device correctly

"git bisect visualize" returns:

commit 774ac8b7eff69e0786970157de2157e68b22f456
Author: Zhang Rui <rui.zhang>
Date:   Fri Oct 30 16:31:47 2015 +0800

    Thermal: initialize thermal zone device correctly
    
    commit bb431ba26c5cd0a17c941ca6c3a195a3a6d5d461 upstream.
    
    After thermal zone device registered, as we have not read any
    temperature before, thus tz->temperature should not be 0,
    which actually means 0C, and thermal trend is not available.
    In this case, we need specially handling for the first
    thermal_zone_device_update().
    
    Both thermal core framework and step_wise governor is
    enhanced to handle this. And since the step_wise governor
    is the only one that uses trends, so it's the only thermal
    governor that needs to be updated.
    
    Tested-by: Manuel Krause <manuelkrause>
    Tested-by: szegad <szegadlo.pl>
    Tested-by: prash <prash.n.rao>
    Tested-by: amish <ammdispose-arch>
    Tested-by: Matthias <morpheusxyz123>
    Reviewed-by: Javi Merino <javi.merino>
    Signed-off-by: Zhang Rui <rui.zhang>
    Signed-off-by: Chen Yu <yu.c.chen>
    Signed-off-by: Greg Kroah-Hartman <gregkh>

FedBisect also requires the following dependencies (from a base install of Fedora):
openssl-devel
bc
gcc
m4
net-tools

Thanks for using the bisect scripts. They are still a work in progress so I will update for the Found your commit.

What you found is a good candidate for causing a perf drop. It's first in a series so I can't revert it by itself to test. Can you test http://koji.fedoraproject.org/koji/taskinfo?taskID=13369275 when it finishes ? This reverts the thermal series on top of 4.4.5.

I just tested kernel-4.4.5-300.perfdropreverts.fc23.x86_64, and it works great, no performance issues experienced.

Thanks for all the work on this!

The upstream developers want to know if this happens on 4.5 as well, can you test this? http://koji.fedoraproject.org/koji/buildinfo?buildID=744823

kernel-4.5.0-300.fc24.x86_64 also experiences the same significant performance decrease.

In addition, with this kernel, my system completely freezes any time X is launched, I had to start in "run level" 3.  This is unrelated to the original problem, so I'm not going to look into this further, just interesting.

Thanks for testing. More requests for info:

the output of "grep . /sys/class/thermal/*/*" on working and good kernel (preferably using the 4.4.5 scratch build I gave as a test since that will be fairly close)

On the bad kernel, the output of grep . /sys/devices/system/cpu/intel_pstate/*

Do you still see the problem if you set /sys/class/thermal/thermal_zone*/mode to "disabled"

Created attachment 1137332 [details]
prash-openssl-speed

I can't reproduce the problem on my system.

For reference, I'm one of the original reporters of the bug in the handling of the thermal subsystem. My affected device is a HP ProBook 4410s laptop running Archlinux. The patches by Rui Zhang and Chen Yu fixed my problem, and I have been running patched kernels for a year now. I did not notice a drop in performance at any time.

For this bug report, I ran "openssl speed" like Jason H. I tested three different kernel versions: 4.1.19(LTS), 4.5.0-rc6-g18558ca, and 4.5.0, the last two of which, include the patches by Rui Zhang and Chen Yu. The 4.5.x kernels are 0.02% slower than the LTS kernels, but for my system, I can't say if they are (1) significant and (2) attributable to these patches.

Seems like these patches are incompatible with newer processors or chipsets.

Laura, I will get you that information tonight.

Prash, since you mentioned Arch, I did test against different versions of the kernel on Arch as well, and the "good" baseline was a little slower than the comparable Fedora kernel, but nothing critical.  While using the 4.4.4 kernel from Arch, performance dropped by around 25%.  Not as a significant decrease as in Fedora (~%82 percent drop), but still a noticeable decrease for me.

I'm sure someone smarter than I will find a way to solve both our issues!

Created attachment 1137504 [details]
/sys/devices/system/cpu/intel_pstate for 4.4.4-301.fc23.x86_64 (slow)

Output of "grep . /sys/devices/system/cpu/intel_pstate/*" for slow kernel 4.4.4.

Created attachment 1137505 [details]
/sys/devices/system/cpu/intel_pstate for 4.4.5-300.perfdropreverts.fc23.x86_64 (fast)

Output of "grep . /sys/devices/system/cpu/intel_pstate/*" for working kernel

Created attachment 1137507 [details]
/sys/class/thermal for 4.4.4-301.fc23.x86_64 (slow)

Output of "grep . /sys/class/thermal/*/*" for slow kernel.

Created attachment 1137508 [details]
/sys/class/thermal for 4.4.5-300.perfdropreverts.fc23.x86_64

Output of "grep . /sys/class/thermal/*/*" command for working kernel.

I have attached the requested outputs of /sys/class/thermal and /sys/devices/system/cpu/intel_pstate.

Setting /sys/class/thermal/thermal_zone0/mode to "disabled" had no effect to the performance issues on the 4.4.4 kernel.  My thermal_zone1 does not have a "mode" parameter to set.

*** Bug 1317147 has been marked as a duplicate of this bug. ***

The patch authors gave a fix that someone else confirmed fixes the performance issue for them. The output from the thermal files here shows the same trip point weirdness so it looks like the same issue. I pulled in the patch to the tree. It should be available when 4.4.7 comes out (later this week or next).

Thanks again for reporting and following up.

kernel-4.4.6-301.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-7e602c0e5e

kernel-4.4.6-201.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2016-ed5110c4bb

(In reply to Fedora Update System from comment #35)
> kernel-4.4.6-301.fc23 has been submitted as an update to Fedora 23.
> https://bodhi.fedoraproject.org/updates/FEDORA-2016-7e602c0e5e

I have tested kernel-4.4.6-301.fc23 x86_64, and it fixes my performance issues.

Thanks!

kernel-4.4.6-201.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-ed5110c4bb

kernel-4.4.6-301.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-7e602c0e5e

kernel-4.5.0-302.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-81fd1b03aa

kernel-4.5.0-302.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.

kernel-4.5.0-302.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.

kernel-4.4.6-301.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.

kernel-4.4.6-201.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.