Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 741325
Summary: | ARM fc14 kernels does not provide hardware perf counter support | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | William Cohen <wcohen> | ||||||
Component: | kernel | Assignee: | Dennis Gilmore <dennis> | ||||||
Status: | CLOSED RAWHIDE | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||
Severity: | medium | Docs Contact: | |||||||
Priority: | unspecified | ||||||||
Version: | 17 | CC: | aph, fche, gansalmon, hhorak, itamar, jonathan, kernel-maint, madhu.chinakonda, pbrobinson | ||||||
Target Milestone: | --- | Keywords: | Reopened | ||||||
Target Release: | --- | ||||||||
Hardware: | arm9 | ||||||||
OS: | Linux | ||||||||
Whiteboard: | ARM | ||||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||||
Doc Text: | Story Points: | --- | |||||||
Clone Of: | Environment: | ||||||||
Last Closed: | 2012-11-15 21:41:09 UTC | Type: | --- | ||||||
Regression: | --- | Mount Type: | --- | ||||||
Documentation: | --- | CRM: | |||||||
Verified Versions: | Category: | --- | |||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||
Embargoed: | |||||||||
Bug Depends On: | |||||||||
Bug Blocks: | 245418, 773116 | ||||||||
Attachments: |
|
Description
William Cohen
2011-09-26 15:22:15 UTC
The kernel on the trimslice seems to provide access to the performance hardware: # uname -a Linux trimslice 2.6.38.3-trimslice-1.01-01637-gc2b2d3e #49 SMP PREEMPT Thu Jul 14 19:18:14 IDT 2011 armv7l armv7l armv7l GNU/Linux Have on the machine: linux-tools install linux-tools-2.6.38-12 install linux-tools-common install linux-firmware install linux-image-trimslice install Have to work around some versioning issues in the perf wrapper and run /usr/bin/perf_2.6.38-12 directly. However, it does provide events information: # perf_2.6.38-12 stat ls etc logs sbin usr Performance counter stats for 'ls': 25,005 cache-misses # 0.804 M/sec 2,801 cache-references # 0.090 M/sec 1,54,875 branch-misses # 46.538 % 3,32,791 branches # 10.697 M/sec 29,16,394 instructions # 0.440 IPC 66,29,814 cycles # 213.109 M/sec 203 page-faults # 0.007 M/sec 0 CPU-migrations # 0.000 M/sec 8 context-switches # 0.000 M/sec 31.110000 task-clock-msecs # 0.803 CPUs 0.038732000 seconds time elapsed Looks like should see what differences there are between the Fedora14 kernel and the this trimslice kernel The ubuntu trimslice kernel is here: http://gitorious.org/trimslice-kernel I added this patch to f15/f16/master. The f14 branch is stuck on 2.6.35 for the rest of its life, and lacks all the arm additions. It seems it's still not working in 2.6.41.6-1.fc15. I don't have a real arm machine so I've created a simple dummy package and tried to build it in arm-koji: http://arm.koji.fedoraproject.org/koji/taskinfo?taskID=267809 There is the following test in %check: perf stat ls || dmesg | tail and the relevant part of build.log is: + perf stat ls Error: open_counter returned with 19 (No such device). /bin/dmesg may provide additional information. Fatal: Not all events could be opened. + tail + dmesg [1089532.581085] hw perfevents: unable to reserve pmu Do I miss something? Testing this on koji is likely to fail. There are no guarantees that the build system has a kernel that support perf (or other feature kernel-specific features). Also the perf support is very specific to the processor implementation. This koji build was done on omap machine, while the original bug report was on a tegra2 processor. Really need to test on tegra2 machine to verify that fedora kernel are fixed. Another data point, the trimslice machine original used to create this report is running a stock 3.2.0+ git tree kernel. $ uname -a Linux fedora-arm 3.2.0+ #30 SMP PREEMPT Wed Jan 18 12:00:36 EST 2012 armv7l armv7l armv7l GNU/Linux In dmesg see: [ 0.140076] hw perfevents: enabled with ARMv7 Cortex-A9 PMU driver, 7 counters available However, for perf (both rpm installed and one from locally built kernel) I get the following: $ ./perf stat ls Error: open_counter returned with 19 (No such device). /bin/dmesg may provide additional information. Fatal: Not all events could be opened. The git commit for the kernel (listed below) only enables the building of the perf package. It doesn't actually do anything to address the problem with accessing the performance monitoring unit in the kernel. commit bf976a17c4b2c2b164f6413d52ffbdab09a26d1a Author: Dave Jones <davej> Date: Tue Nov 1 14:08:42 2011 -0400 allow building the perf rpm for ARM (rhbz 741325) (In reply to comment #5) > The git commit for the kernel (listed below) only enables the building of the > perf package. It doesn't actually do anything to address the problem with > accessing the performance monitoring unit in the kernel. According comment #2 it should be possible to have performance counter support in 2.6.x. I'd like to see this working because of bug #773116, so re-opening (or should I open a new bz for that?). the 2.6.x on fedora arm is actually all 3.x based. F14 is EOL and not really supported. the kernel in koji for f14 is backported from F15. I suggest you use f15 arm at this point. we are rappidly getting rawhide up. f15 will have a short life also. normal release lifes and schedules should follow after that. Since current F15 kernel still doesn't fix this bug, I've changed version to F15. Let's keep this open until at least F16 or Rawhide with performance counter support is ready. perf seems to run just fine on a armv5tel F-15 install (currently even running a F-14 kernel too), see output below. Looking at the current F-15 kernel it seems we're building the perf package for softfp but not hardfp: http://arm.koji.fedoraproject.org/koji/buildinfo?buildID=41164 [root@fedora-arm ~]# rpm -q kernel-omap perf kernel-omap-2.6.40.4-6.fc15.armv7l kernel-omap-2.6.41.6-1.fc15.armv7l perf-2.6.41.6-1.fc15.armv5tel [root@fedora-arm ~]# uname -a Linux fedora-arm 2.6.40.3-0.fc14.armv7l.omap #1 SMP PREEMPT Mon Aug 22 12:46:51 EDT 2011 armv7l armv7l armv7l GNU/Linux [root@fedora-arm ~]# perf stat ls Performance counter stats for 'ls': 9.002685 task-clock # 0.853 CPUs utilized 0 context-switches # 0.000 M/sec 0 CPU-migrations # 0.000 M/sec 198 page-faults # 0.022 M/sec 3000471 cycles # 0.333 GHz 0 stalled-cycles-frontend # 0.00% frontend cycles idle 0 stalled-cycles-backend # 0.00% backend cycles idle 1319528 instructions # 0.44 insns per cycle 171539 branches # 19.054 M/sec <not counted> branch-misses 0.010558721 seconds time elapsed (In reply to comment #9) > Looking at the current F-15 kernel it seems we're building the perf package for > softfp but not hardfp: > > http://arm.koji.fedoraproject.org/koji/buildinfo?buildID=41164 I'm still trying to understand it. Does it mean it is possible to read cycle count in userspace with that kernel? And my dummy test (see comment #4) means we are not able to get cycle count only in koji? As mentioned the builder might be using a different kernel that the distribution. The test didn't include the kernel version that being used by the builder, so it isn't clear which specific kernel was being used for this experiment on the build system. The tail end of a backtrace is in the output there: + dmesg [367699.273895] [<c0053524>] (__do_softirq+0x5c/0x244) from [<c0053b7c>] (irq_exit+0x90/0x98) [367699.282592] [<c0053b7c>] (irq_exit+0x90/0x98) from [<c000e4c8>] (handle_IRQ+0x50/0xac) [367699.291046] [<c000e4c8>] (handle_IRQ+0x50/0xac) from [<c04e7054>] (__irq_usr+0x34/0x100) [367699.299682] ---[ end trace e2d689825ccd843e ]--- kernel-omap and kernel-tegra2 are different sub packages. It is quite possible that perf could work in one but not in another because of differences in the config or code path being used. I installed the tegra kernel from: http://arm.koji.fedoraproject.org/koji/buildinfo?buildID=41164 And followed the setup instructions at: http://fedoraproject.org/wiki/Architectures/ARM/TrimSlicePRO Tried the same experiment and things do not work on the trimslice [wcohen@fedora-arm ~]$ uname -a Linux fedora-arm 2.6.41.6-1.fc15.armv7l.tegra #1 SMP PREEMPT Sat Dec 24 22:32:17 UTC 2011 armv7l armv7l armv7l GNU/Linux [wcohen@fedora-arm ~]$ perf stat ls Error: open_counter returned with 19 (No such device). /bin/dmesg may provide additional information. Fatal: Not all events could be opened. In dmesg see: [ 0.130079] hw perfevents: enabled with ARMv7 Cortex-A9 PMU driver, 7 counter s available Then later: [ 202.117626] hw perfevents: unable to reserve pmu So there is something broken for the tegra 2 use of the performance monitoring unit. Which omap processor was used in comment#9? I wonder if the difference between omap working and tegra not is Cortex-A8 vs Cortex-A9: http://developer.nvidia.com/archived-tegra-forums/forum/oprofile-linux It would be really useful to know which version of the PMU is working on the omap. What was the output in dmesg? (In reply to comment #11) > As mentioned the builder might be using a different kernel that the > distribution. The test didn't include the kernel version that being used by the > builder, so it isn't clear which specific kernel was being used for this > experiment on the build system. The tail end of a backtrace is in the output > there: Same kernel but armv5tel on omap > kernel-omap and kernel-tegra2 are different sub packages. It is quite possible > that perf could work in one but not in another because of differences in the > config or code path being used. I installed the tegra kernel from: Yes, > http://arm.koji.fedoraproject.org/koji/buildinfo?buildID=41164 If you look @ that kernel there's no perf package for armv7hl, the armv5tel one won't work. I'm looking to get that fixed. Once I have it fixed I'll test it on my TS running the 7hl kernel/perf > Tried the same experiment and things do not work on the trimslice where did you get the perf package from? (In reply to comment #12) > Which omap processor was used in comment#9? I wonder if the difference between > omap working and tegra not is Cortex-A8 vs Cortex-A9: Its a beagleboard XM > http://developer.nvidia.com/archived-tegra-forums/forum/oprofile-linux
"It appears the oprofile code in the tegra/android kernel is quite different from mainline linux and not compatible with the normal oprofile userspace tools. "
We're using the mainline linux kernel, not an android kernel.
My trimslice is running $ rpm -q kernel-tegra perf kernel-tegra-2.6.41.6-1.fc15.armv7l perf-2.6.41.6-1.fc15.armv5tel The perf armv5tel appears to work for the software events with the arm7l kernel. It is just the the hardware events that do not appear to work: $ perf stat -e task-clock -e minor-faults -e major-faults -e context-switches ls autoconf-vm-area-pte.c papi-4.2.0-1.fc17.src.rpm ioblktime2.stp perf.data ioblktime.stp transcheck.diff kernel-2.6.41.6-1.fc15.src.rpm valgrind-3.6.1-6.fc16.src.rpm Performance counter stats for 'ls': 4.437000 task-clock # 0.260 CPUs utilized 0 minor-faults # 0.000 M/sec 0 major-faults # 0.000 M/sec 23 context-switches # 0.005 M/sec 0.017036764 seconds time elapsed $ perf stat -e cycles ls Error: open_counter returned with 19 (No such device). /bin/dmesg may provide additional information. Fatal: Not all events could be opened. The beagle board xm is cortex-a8, so there are definitely difference in code being run. Ran a couple very simple sytsemtap script to get a better idea what is going on. $ stap -e 'probe kernel.function("reserve_pmu").return {printf("%s %s 0x%x\n", pn(), $$parms$, $return)}' -c 'perf stat ls' Error: open_counter returned with 19 (No such device). /bin/dmesg may provide additional information. Fatal: Not all events could be opened. Warning: child process exited with status 128 kernel.function("reserve_pmu@arch/arm/kernel/pmu.c:115").return type=0 0xffffffed Warning: /usr/bin/staprun exited with status: 1 Looks like things are going wrong in reserve_pmu(). Return values of -19, -ENODEV. It looks like pmu_devices[type] is not getting initialized. Maybe pmu_register() and register_pmu_driver() are never called. http://lxr.linux.no/#linux+v3.1.5/arch/arm/kernel/pmu.c#L29 Is there something in the dmesg output like "registered new PMU device of type ..." for the beagle board (or other ARM system with working PMU support)? Nothing like that in the trimslice dmesg output. A little experiment to see what function are getting called using the para-callgraph.stp example. For a working software event see: stap /usr/share/doc/systemtap-1.6/examples/general/para-callgraph.stp 'kernel.function("*@arch/arm/kernel/perf*.c")' -c 'perf stat -e cpu-clock ls' autoconf-vm-area-pte.c papi-4.2.1-0.20120209.el6.src.rpm ioblktime2.stp perf.data ioblktime.stp transcheck.diff kernel-2.6.41.6-1.fc15.src.rpm valgrind-3.6.1-6.fc16.src.rpm Performance counter stats for 'ls': 3.705000 cpu-clock 0.004748231 seconds time elapsed 0 perf(8363):->armpmu_event_init event=0xede6ac00 0 perf(8363): ->armv7_a9_map_event event=0xede6ac00 0 perf(8363): ->map_cpu_event event=0xede6ac00 event_map=0xc0483e34 cache_map=0xc0483e5c raw_event_mask=0xff 0 perf(8363): <-map_cpu_event return=0xfffffffffffffffe 0 perf(8363): <-armv7_a9_map_event return=0xfffffffffffffffe 0 perf(8363):<-armpmu_event_init return=0xfffffffffffffffe For a non-working hardware event (cycles) see (the 0xff returned by map_cpu_event looks like the expected value according to armv7_a9_perf_map[]) :: stap /usr/share/doc/systemtap-1.6/examples/general/para-callgraph.stp 'kernel.function("*@arch/arm/kernel/perf*.c")' -c 'perf stat -e cycles ls' Error: open_counter returned with 19 (No such device). /bin/dmesg may provide additional information. Fatal: Not all events could be opened. Warning: child process exited with status 128 0 perf(8351):->armpmu_event_init event=0xede6ac00 0 perf(8351): ->armv7_a9_map_event event=0xede6ac00 0 perf(8351): ->map_cpu_event event=0xede6ac00 event_map=0xc0483e34 cache_map=0xc0483e5c raw_event_mask=0xff 0 perf(8351): <-map_cpu_event return=0xff 0 perf(8351): <-armv7_a9_map_event return=0xff 0 perf(8351):<-armpmu_event_init return=0xffffffffffffffed WARNING: /media/greatplains/wcohen/systemtap_write/install/bin/staprun exited with status: 1 Pass 5: run failed. Try again with another '--vp 00001' option. Never see it call to __hw_perf_event_init(event) with: stap -e 'probe kernel.function("__hw_perf_event_init") {printf("%s %s\n",pn(), $event$)}' -c 'perf stat ls' Error: open_counter returned with 19 (No such device). /bin/dmesg may provide additional information. Fatal: Not all events could be opened. Warning: child process exited with status 128 WARNING: /media/greatplains/wcohen/systemtap_write/install/bin/staprun exited with status: 1 Pass 5: run failed. Try again with another '--vp 00001' option. So things look to be going wrong in armpmu_reserve_hardware() Suspect armpmu->plat_device is never set so in armpmu_reserve_hardware() end up running 405 if (!pmu_device) 406 return -ENODEV; That would explain the behavior. This line of code looks like it is suppose to initialize that plat_device but isn't http://lxr.linux.no/#linux+v3.2.5/arch/arm/kernel/perf_event.c#L646 Looks like some information is missing from the device tree for tegra*.dts. In upstream linux kernel see arm/arch/boot/dts/highbank.dts see the following: cpus { ... soc { #address-cells = <1>; #size-cells = <1>; compatible = "simple-bus"; interrupt-parent = <&intc>; ranges; ... pmu { compatible = "arm,cortex-a9-pmu"; interrupts = <0 76 4 0 75 4 0 74 4 0 73 4>; }; For the hardware perf events to work on various arm implementations for newer linux kernel they are going to need similar device tree entries. Some mentions of requiring this information in the device tree in http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/devicetree/bindings/arm/pmu.txt; make sure your f15 system is fully updated. the f15 update fedora kernel added perf on armv7hl I have been investigating this with the upstream kernel git tree (git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git). The only device tree description that seems like it has a chance of working for hardware performance monitoring events is the highbank.dts one. Nothing else has a descriptions for the performance monitoring unit (pmu) descriptions. Took a look at kernel-2.6.42.2-1.fc15.src.rpm sources and there doesn't seem to be any changes to address this. The user-space perf package is not the problem. perf is able to measure software events without problem. There is some code in arch/arm/mach-tegra/devices.c related to the pmu, but there doesn't seem to be any use of that in arch/arm/mach-tegra/devices.c. The board-seaboard.c includes an entry. Looks like there should be a similar entry in board-trimslice.c. Last i looked none of the framebuffer or Power management code for the trimslice is in the upstream kernel. its all still only in compulabs git repo. so we dont have any of it supported. since it needs to move to upstream first. Created attachment 567210 [details]
Enable hardware performance monitoring support on trimslice
A one line patch to fix up support of performance monitoring on trimslice. Compiled kernel now allows:
$ perf stat ls .
boot.scr prelink-0.4.5-4.fc17.src.rpm
kernel-tegra-perf-dtb.patch prelink-0.4.6-4.fc18.src.rpm
Performance counter stats for 'ls .':
4.245000 task-clock # 0.848 CPUs utilized
0 context-switches # 0.000 M/sec
0 CPU-migrations # 0.000 M/sec
207 page-faults # 0.049 M/sec
4228072 cycles # 0.996 GHz
788958 stalled-cycles-frontend # 18.66% frontend cycles idle
3279599 stalled-cycles-backend # 77.57% backend cycles idle
1438908 instructions # 0.34 insns per cycle
# 2.28 stalled cycles per insn
143618 branches # 33.832 M/sec
70850 branch-misses # 49.33% of all branches
0.005006798 seconds time elapsed
There are people that are wanting to look at performance issues on trimslice boxes. Without this fixed none of the performance tools (perf/pcl, papi, or oprofile) will work making it more difficult for people address performance issues on the upcoming fedora 17 for arm. (In reply to comment #26) > There are people that are wanting to look at performance issues on trimslice > boxes. Without this fixed none of the performance tools (perf/pcl, papi, or > oprofile) will work making it more difficult for people address performance > issues on the upcoming fedora 17 for arm. Has the one line patch been pushed to mainline kernel? There is a request to put that into the upstream kernels. Thread at: http://www.spinics.net/lists/arm-kernel/msg162913.html However, it doesn't appear to be in yet. Moving to F-17 as still an issue there and it's the only supported ARM release. I need to work out how we can better get patches upstreamed Will can you test F-18 with the 3.6.1 kernel and the new Trimslice uboot that supports DeviceTree. With that it should work I was able to set up kernel-tegra-3.6.1-2.fc18.armv7hl on f17 to boot with device trees and able to get data out of perf. [wcohen@fedora-arm src]$ uname -a Linux fedora-arm 3.6.1-2.fc18.armv7hl.tegra #1 SMP Sat Oct 13 19:02:46 EDT 2012x [wcohen@fedora-arm src]$ ls -d /proc/device-tree/ /proc/device-tree/ [wcohen@fedora-arm src]$ perf stat true Performance counter stats for 'true': 5.366000 task-clock # 0.367 CPUs utilized 1 context-switches # 0.186 K/sec 0 CPU-migrations # 0.000 K/sec 73 page-faults # 0.014 M/sec 1140567 cycles # 0.213 GHz 199019 stalled-cycles-frontend # 17.45% frontend cycles idle 866886 stalled-cycles-backend # 76.00% backend cycles idle 400278 instructions # 0.35 insns per cycle # 2.17 stalled cycles per insn 43171 branches # 8.045 M/sec 22859 branch-misses # 52.95% of all branches 0.014617677 seconds time elapsed The performance monitoring hardware works when device trees used on trimslice, so closing this bug. |