Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 455595

Summary: kernel-2.6.27-0.144.rc0.git2.fc10.i686 and later seems hang during udev
Product: [Fedora] Fedora Reporter: Tom London <selinux>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: low    
Version: rawhideCC: gnafu_the_great, joshuacov, krh, olivares14031, poelstra, redhat-bugzilla, shawn.starr, yaneti
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2008-10-20 18:58:19 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 466414    
Attachments:
Description Flags
/var/log/messages of boot with udevdebug=1
none
picture of frozen system - 2 penguins + udev + frozen cursor
none
photo of screen w/ strace output of hung udev ....
none
Picture of screen when hung: booted with "debug ignore_loglevel"
none
linux-2.6-x86-32-amd-c1e-force-timer-broadcast-late.patch none

Description Tom London 2008-07-16 14:26:28 UTC
Description of problem:
Installing this morning's rawhide kernel and rebooting, I get a hang with a
frozen spinfinity screen (with no spinner).

Removing 'rhgb quiet' from the boot line allows the system to boot normally.

When I do this, I get no fsck messages, so I'm guessing it hangs before udev, etc.

Version-Release number of selected component (if applicable):
plymouth-0.5.0-5.fc10.i386
kernel-2.6.27-0.144.rc0.git2.fc10.i686
mkinitrd-6.0.55-1.fc10.i386
nash-6.0.55-1.fc10.i386

How reproducible:
Every time

Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

Comment 1 Tom London 2008-07-16 16:19:06 UTC
This appears to be a kernel problem:  I got it to hang during udev scan when
booting in text mode (e.g., with rhgb/quiet removed from the boot line).

I'll redirect this to kernel.....

Comment 2 Tom London 2008-07-16 16:22:47 UTC
A bit more info:

I can get this to boot sometimes by turning off 'rhgb quiet' from boot.

But I can also get this to hang during udev even in text mode.

I tried booting with 'udevdebug=1' (sorry if that is not completely correct),
and I got voluminous output, some red, some green.  The system did make it past
udev and did boot to gdm login, but I could not successfully login.

Comment 3 Tom London 2008-07-16 16:57:22 UTC
Seems to fail the same way with kernel-2.6.27-0.148.rc0.git2.fc10.i686 from koji.

Additional info I can provide?

Comment 4 Tom London 2008-07-16 18:48:55 UTC
More info on my system: Thinkpad X60:

[root@localhost ~]# lspci
00:00.0 Host bridge: Intel Corporation Mobile 945GM/PM/GMS, 943/940GML and 945GT
Express Memory Controller Hub (rev 03)
00:02.0 VGA compatible controller: Intel Corporation Mobile 945GM/GMS,
943/940GML Express Integrated Graphics Controller (rev 03)
00:02.1 Display controller: Intel Corporation Mobile 945GM/GMS/GME, 943/940GML
Express Integrated Graphics Controller (rev 03)
00:1b.0 Audio device: Intel Corporation 82801G (ICH7 Family) High Definition
Audio Controller (rev 02)
00:1c.0 PCI bridge: Intel Corporation 82801G (ICH7 Family) PCI Express Port 1
(rev 02)
00:1c.1 PCI bridge: Intel Corporation 82801G (ICH7 Family) PCI Express Port 2
(rev 02)
00:1c.2 PCI bridge: Intel Corporation 82801G (ICH7 Family) PCI Express Port 3
(rev 02)
00:1c.3 PCI bridge: Intel Corporation 82801G (ICH7 Family) PCI Express Port 4
(rev 02)
00:1d.0 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI
Controller #1 (rev 02)
00:1d.1 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI
Controller #2 (rev 02)
00:1d.2 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI
Controller #3 (rev 02)
00:1d.3 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI
Controller #4 (rev 02)
00:1d.7 USB Controller: Intel Corporation 82801G (ICH7 Family) USB2 EHCI
Controller (rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev e2)
00:1f.0 ISA bridge: Intel Corporation 82801GBM (ICH7-M) LPC Interface Bridge
(rev 02)
00:1f.1 IDE interface: Intel Corporation 82801G (ICH7 Family) IDE Controller
(rev 02)
00:1f.2 SATA controller: Intel Corporation 82801GBM/GHM (ICH7 Family) SATA AHCI
Controller (rev 02)
00:1f.3 SMBus: Intel Corporation 82801G (ICH7 Family) SMBus Controller (rev 02)
02:00.0 Ethernet controller: Intel Corporation 82573L Gigabit Ethernet Controller
03:00.0 Network controller: Intel Corporation PRO/Wireless 3945ABG Network
Connection (rev 02)
15:00.0 CardBus bridge: Ricoh Co Ltd RL5c476 II (rev b4)
15:00.1 FireWire (IEEE 1394): Ricoh Co Ltd R5C552 IEEE 1394 Controller (rev 09)
15:00.2 SD Host controller: Ricoh Co Ltd R5C822 SD/SDIO/MMC/MS/MSPro Host
Adapter (rev 18)
[root@localhost ~]# 

Comment 5 Tom London 2008-07-16 18:55:15 UTC
Created attachment 311982 [details]
/var/log/messages of boot with udevdebug=1

I'm located the /var/log/messages for the run I made with udevdebug=1

Comment 6 Tom London 2008-07-16 19:22:59 UTC
Created attachment 311986 [details]
picture of frozen system - 2 penguins + udev + frozen cursor

Not sure its helpful, but took this picture of screen when frozen.

Penguins still there....

Works fine with kernel-2.6.26-138.fc10.i686

Comment 7 Tom London 2008-07-17 13:58:24 UTC
This seems intermittant....

I can occasionally get it to complete booting with 0.148 (like 1 in 4 or 5).

When it does boot, I see no issues and no error messages of interest.

When it doens't boot, always seems to hang the same way: before udev probing
completes with 2 penguins still displayed.

On a hunch, I found a BIOS update from IBM/Lenovo and applied (2.14->2.17).  No
change, still hangs most boots.

Comment 8 Tom London 2008-07-17 16:13:59 UTC
Similar issue with 0.156.

When I can get it past udev boot scan (about 1 in 4 or 5 boots), I can find no
problems....

Comment 9 Tom London 2008-07-17 18:03:05 UTC
OK.  Digging into this, I instrumented /sbin/start_udev to see where it is freezing.

I sprinkled 'echo -n " 1"' -> 'echo -n " 16"' in the shell commands, and I
consistently get it freezing at 'wait_for_queue $udevtimeout'.  Here's a snippet
of the changes to /sbin/start_udev:

<<<<<SNIP>>>>>
echo -n " 13"

	/sbin/udevadm trigger
	ret=$[$ret + $?]
echo -n " 14"

	wait_for_queue $udevtimeout
	ret=$[$ret + $?]
echo -n " 15"

	wait
	/sbin/udevadm control env STARTUP=
echo -n " 16"
<<<<<SNIP>>>>>

When it freezes, I always get  the freeze after the 'echo -n " 14"' and before
the "15".

I'll add to the instrumentation and report.  Anything known that could be
causing this?

Comment 10 Tom London 2008-07-17 18:35:35 UTC
Adding additional 'echos' to start_udev seems to dramatically increase the
likelihood that the boot will succeed.

Adding the following (last) 'echos' has made this boot 4 times in a row:

wait_for_queue() {
        local timeout=${1:-0}
        local ret=0
echo -n " 14.1 timeout=$timeout"

        if [ $timeout -gt 0 ]; then
echo -n " 14.2"
            /sbin/udevsettle --timeout=$timeout
        else
echo -n " 14.3"
            /sbin/udevsettle
        fi
        ret=$?
        if [ $ret -ne 0 ]; then
                echo -n "Wait timeout. Will continue in the background."
        fi
        return $ret;
}

BTW, timeout is 0 in these runs.

I'm thinking that "left to itself (i.e., without the delays caused by
printing)", the call to "udevsettle" just hangs.

Is this a udev issue?  (Although this code works for 0.138 and earlier kernels).

Comment 11 Tom London 2008-07-18 23:20:44 UTC
Created attachment 312179 [details]
photo of screen w/ strace output of hung udev ....

I managed to capture a "hang in start_udev" and attach a picture of the screen
showing the output of strace of udevsettle (aka 'udevadm settle').

I hope this helps.....

Seems to me that 'udevsettle' can hang somewhere if called "too early".

Specifying a 'udevtimeout' doesn't seem to help.... still hangs.

If I put a 5-8 second sleep in start_udev before calling udevsettle, the system
boots fine.  Strace output of that case shows a bunch of 'nanosleeps()' and
what seems to be polling of /dev/.udev/...queue  (udevqueue?) that always seems
to eventually 'stop'.

Comment 12 Tom London 2008-07-20 01:10:36 UTC
Tracking this down further, it appears to have nothing to do with 'udevsettle'.

I boot typically with 'vga=791', and this has only happens when I boot with
'vga=791'. 

Removing 'vga=791' seems to make the system boot fine.  I have no idea why.....

Booting with 'vga=791', the system hangs before (or as) it changes the video
mode to use all the lines on the screen (and erase the nice 2 penguins).

If the 2 penguins vanish, the system boots fine.  Otherwise, the system hangs
(whether or not 'udevsettle' has been called seems not to matter).

Independent of how I boot, I do see a very quick 'flicker' on the screen during
the 'Starting udev:' time period.

I've booted with 'vga=791' since fc1 or fc2 days..... and this just started
becoming flakey with 2.6.27-0.141.

Does this make any sense?

Comment 13 Tom London 2008-07-21 14:19:34 UTC
I decided to do a 'complete' test of all the valid 'vga=' boot modes for my
Thinkpad X60 running kernel-2.6.27-0.159.rc0.git6.fc10.i686.

The result of trying to boot with 'rhgb quiet vga=0xXXX' shows only 50% of them
working.

I obtained the list of valid vga codes by entering 'vga=799' to grub, and the
kernel graciously listed the valid codes for my system.  I picked the ones that
matched my monitor.

The setup was my Thinkpad in its 'dock' with the lid closed, connected to an
external monitor.  All the below tests were done in a single location, but the
above results are from 2 locations with 2 different monitors.

I tried all valid 1280x1024, 1024x768, 800x600 and 640x408 modes (32-bit, 16-bit
and 8-bit for each) for a total of 12 cases.

Only 6 booted.  Booting without any 'vga=' option also booted.  I believe it is
selecting an 8-bit mode (looks like 640x480 or 800x600 to me), as I don't get
the spiffy 'spinfinity' animation, just the sliding bar animation.

All the test cases appeared to freeze similarly: the animation starts, and then
freezes after about 5-10 seconds.  This is consistent with the above reported
'freezes during "Starting udev:"'.

Here are the results, listed by resolution and then by 'bits':

Mode     Result

0x31B     Booted    (1280x1024x32)
0x31A     Booted    (1280x1024x16)
0x307     Froze     (1280x1024x8)  no Spinfinity

0x318     Froze     (1024x768x32)
0x317     Froze     (1024x768x16)  my old 'vga=791'
0x305     Froze     (1024x768x8)   no Spinfinity

0x315     Froze     (800x600x32)
0x315     Booted    (800x600x16)
0x303     Booted    (800x600x8)    no Spinfinity

0x312     Booted    (640x480x32)
0x311     Booted    (640x480x16)
0x301     Froze     (640x480x8)    no Spinfinity

Thoughts on what this could be?

Suggestions for further testing/information?

Comment 14 Ralf Ertzinger 2008-07-22 14:51:40 UTC
My X60s does not boot with vga=0x318, does _not_ boot without vga, but does boot
with vga=0x312

Comment 15 Tom London 2008-07-22 21:16:16 UTC
Created attachment 312396 [details]
Picture of screen when hung: booted with "debug ignore_loglevel"

I booted with "debug ignore_loglevel" and got this hang.

It doesn't always hang when these additional debug options are set.

The last few things printed relate to iwl3945 and ACPI.

Comment 16 Tom London 2008-07-24 14:43:02 UTC
This continues with kernel-2.6.27-0.173.rc0.git11.fc10.i686

Boots with 'vga=795' and no 'rhgb quiet' about 1 in 3 attempts.

Comment 17 Tom London 2008-07-26 19:56:13 UTC
I continue to have this problem with every kernel up to and include 0.183.

There seems to be a "race" of some sort: most boots, the system hangs during
"Starting udev:"; sometimes it succeeds (about 1 in 3 or 4).  Seems to be
independent of booting in plymouth mode, text mode, vga size, etc.

When booting without plymouth, you can tell if the boot will succeed (and not
hang) if the two nice penguins on the top of the screen (booting with vga in
1024x768 or 1280x1024) vanish.  They typically go away in the middle of the
"Starting udev:" step.  

When the system freezes, it really freezes.... the cursor stops blinking, etc.
The only recovery I've discovered is to hard power cycle.

Ralf (#14) also seems to be seeing this, and he also seems to be running on a
Thinkpad (he, X60s, me X60).

Is this a known problem?  If so, I can stop probing and just wait for the fix.

Otherwise, how can I help provide information to track this down.....

Comment 18 Ralf Ertzinger 2008-07-26 21:52:19 UTC
Tom, since you are on a quite simimar Thinkpad:
can you try to disable wireless using the killswitch at the front of the
laptop before booting?

I remember a similar situation some time ago where the system would hang
during udev startup where disabling the wireless controller would work.
If udev has started you can turn WLAN on again.

Comment 19 Tom London 2008-07-26 22:02:49 UTC
Fails the same regardless of killswitch setting....  Thanks for the suggestion
though....

Comment 20 Tom London 2008-07-30 14:48:58 UTC
Further hacking to /sbin/start_udev seems to indicate that the hang occurs
during the call to "/sbin/udevadm trigger".

Here are the changes I made:
[root@localhost sbin]# diff start_udev start_udev.save
277c277
< 	/sbin/udevadm trigger --verbose
---
> 	/sbin/udevadm trigger
279,287d278
< if strstr "$cmdline" tbldelay; then
< 	itmp=0;
< 	while [ $itmp -lt 10 ]; do
< 	    echo -n "0-$itmp."
< 	    sleep 1
< 	    itmp=$[$itmp + 1]
< 	done
< 	echo -n "done! "
< fi

With these changes, when the boot hangs output stops "in the middle" of output
from "/sbin/udevadm trigger --verbose", and before the 10 second counting loop.

Any ideas who/what/where?  I'm clearly in over my head.....

Comment 21 Ralf Ertzinger 2008-07-30 18:57:42 UTC
adding 'modprobedebug' to the command line makes it boot pretty consistently for
me. That's quite annoying.

Comment 22 Tom London 2008-08-01 13:24:36 UTC
First boot attempts (~10) with kernel-2.6.27-0.205.rc1.git2.fc10.i686 are
positive.  All got past starting udev and all booted successfully!



Comment 23 Tom London 2008-08-01 13:37:44 UTC
Ooops.  Premature.

The 11th froze as before.

Whatever the issue is, 0.205 seems to make it happen less often.

Comment 24 Tom London 2008-08-03 01:21:06 UTC
Problem continues for me with 0.211.

Freezes about 30-50% of the times.

Comment 25 Tom London 2008-08-06 14:06:05 UTC
Continues to freeze as above about 50% of the time with 0.226.

Comment 26 Tom London 2008-08-07 14:31:35 UTC
Freezing continues with kernel-2.6.27-0.237.rc2.fc10.i686 about 50% of boots as above.

As per #21, adding 'modprobedebug' to the boot line appears to increase the "successful boot" rate.

Only recovery: hard power cycle.

Comment 27 Antonio A. Olivares 2008-08-08 19:19:36 UTC
This problem affects me as well and it is the first time I encounter it with kernel 2.6.27-0.238.rc2.fc10.i686

Success rate is 1/2.  It hangs the first time and upon a soft reboot it works.

Comment 28 Gideon Mayhak 2008-08-11 23:35:11 UTC
I'm having similar issues with the Alpha-release kernel up to and including the latest (244).  Here is the output of my lspci:

00:00.0 RAM memory: nVidia Corporation C51 Host Bridge (rev a2)
00:00.1 RAM memory: nVidia Corporation C51 Memory Controller 0 (rev a2)
00:00.2 RAM memory: nVidia Corporation C51 Memory Controller 1 (rev a2)
00:00.3 RAM memory: nVidia Corporation C51 Memory Controller 5 (rev a2)
00:00.4 RAM memory: nVidia Corporation C51 Memory Controller 4 (rev a2)
00:00.5 RAM memory: nVidia Corporation C51 Host Bridge (rev a2)
00:00.6 RAM memory: nVidia Corporation C51 Memory Controller 3 (rev a2)
00:00.7 RAM memory: nVidia Corporation C51 Memory Controller 2 (rev a2)
00:02.0 PCI bridge: nVidia Corporation C51 PCI Express Bridge (rev a1)
00:03.0 PCI bridge: nVidia Corporation C51 PCI Express Bridge (rev a1)
00:05.0 VGA compatible controller: nVidia Corporation MCP51 PCI-X GeForce Go 6100 (rev a2)
00:09.0 RAM memory: nVidia Corporation MCP51 Host Bridge (rev a2)
00:0a.0 ISA bridge: nVidia Corporation MCP51 LPC Bridge (rev a3)
00:0a.1 SMBus: nVidia Corporation MCP51 SMBus (rev a3)
00:0a.3 Co-processor: nVidia Corporation MCP51 PMU (rev a3)
00:0b.0 USB Controller: nVidia Corporation MCP51 USB Controller (rev a3)
00:0b.1 USB Controller: nVidia Corporation MCP51 USB Controller (rev a3)
00:0d.0 IDE interface: nVidia Corporation MCP51 IDE (rev f1)
00:0e.0 IDE interface: nVidia Corporation MCP51 Serial ATA Controller (rev f1)
00:10.0 PCI bridge: nVidia Corporation MCP51 PCI Bridge (rev a2)
00:10.1 Audio device: nVidia Corporation MCP51 High Definition Audio (rev a2)
00:14.0 Bridge: nVidia Corporation MCP51 Ethernet Controller (rev a3)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
03:00.0 Network controller: Broadcom Corporation BCM4311 802.11b/g WLAN (rev 02)
[root@gidux-laptop-rawhide ~]# lspci
00:00.0 RAM memory: nVidia Corporation C51 Host Bridge (rev a2)
00:00.1 RAM memory: nVidia Corporation C51 Memory Controller 0 (rev a2)
00:00.2 RAM memory: nVidia Corporation C51 Memory Controller 1 (rev a2)
00:00.3 RAM memory: nVidia Corporation C51 Memory Controller 5 (rev a2)
00:00.4 RAM memory: nVidia Corporation C51 Memory Controller 4 (rev a2)
00:00.5 RAM memory: nVidia Corporation C51 Host Bridge (rev a2)
00:00.6 RAM memory: nVidia Corporation C51 Memory Controller 3 (rev a2)
00:00.7 RAM memory: nVidia Corporation C51 Memory Controller 2 (rev a2)
00:02.0 PCI bridge: nVidia Corporation C51 PCI Express Bridge (rev a1)
00:03.0 PCI bridge: nVidia Corporation C51 PCI Express Bridge (rev a1)
00:05.0 VGA compatible controller: nVidia Corporation MCP51 PCI-X GeForce Go 6100 (rev a2)
00:09.0 RAM memory: nVidia Corporation MCP51 Host Bridge (rev a2)
00:0a.0 ISA bridge: nVidia Corporation MCP51 LPC Bridge (rev a3)
00:0a.1 SMBus: nVidia Corporation MCP51 SMBus (rev a3)
00:0a.3 Co-processor: nVidia Corporation MCP51 PMU (rev a3)
00:0b.0 USB Controller: nVidia Corporation MCP51 USB Controller (rev a3)
00:0b.1 USB Controller: nVidia Corporation MCP51 USB Controller (rev a3)
00:0d.0 IDE interface: nVidia Corporation MCP51 IDE (rev f1)
00:0e.0 IDE interface: nVidia Corporation MCP51 Serial ATA Controller (rev f1)
00:10.0 PCI bridge: nVidia Corporation MCP51 PCI Bridge (rev a2)
00:10.1 Audio device: nVidia Corporation MCP51 High Definition Audio (rev a2)
00:14.0 Bridge: nVidia Corporation MCP51 Ethernet Controller (rev a3)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
03:00.0 Network controller: Broadcom Corporation BCM4311 802.11b/g WLAN (rev 02)

The only way I can get the system to boot it to add acpi=off, but then my wireless doesn't work and I get no power-saving features.  This also was the case with the 64-bit installer (I'm currently running 32-bit).

Comment 29 Shawn Starr 2008-08-29 18:30:09 UTC
I can confirm this also in the newest kernel 2.6.27-0.287.rc4.git7.fc10.

I have a ThinkPad T42, when udev starts up, the hard disk light goes solid and no activity further happens. Rebooting the system will get things working again.

This is with 'text' mode (as kms does not work yet for rv350).

This happens very infrequently I might add.

Comment 30 Joshua Covington 2008-09-02 00:29:47 UTC
I had a similar problem with my acer aspire.
here is the lspci -n
00:00.0 Host bridge [0600]: ATI Technologies Inc RS480 Host Bridge [1002:5950] (rev 10)
00:01.0 PCI bridge [0604]: ATI Technologies Inc RS480 PCI Bridge [1002:5a3f]
00:04.0 PCI bridge [0604]: ATI Technologies Inc RS480 PCI Bridge [1002:5a36]
00:05.0 PCI bridge [0604]: ATI Technologies Inc RS480 PCI Bridge [1002:5a37]
00:06.0 PCI bridge [0604]: ATI Technologies Inc RS480 PCI Bridge [1002:5a38]
00:07.0 PCI bridge [0604]: ATI Technologies Inc RS480 PCI Bridge [1002:5a39]
00:12.0 IDE interface [0101]: ATI Technologies Inc IXP SB400 Serial ATA Controller [1002:4379] (rev 80)
00:13.0 USB Controller [0c03]: ATI Technologies Inc IXP SB400 USB Host Controller [1002:4374] (rev 80)
00:13.1 USB Controller [0c03]: ATI Technologies Inc IXP SB400 USB Host Controller [1002:4375] (rev 80)
00:13.2 USB Controller [0c03]: ATI Technologies Inc IXP SB400 USB2 Host Controller [1002:4373] (rev 80)
00:14.0 SMBus [0c05]: ATI Technologies Inc IXP SB400 SMBus Controller [1002:4372] (rev 83)
00:14.1 IDE interface [0101]: ATI Technologies Inc IXP SB400 IDE Controller [1002:4376] (rev 80)
00:14.2 Audio device [0403]: ATI Technologies Inc IXP SB4x0 High Definition Audio Controller [1002:437b] (rev 01)
00:14.3 ISA bridge [0601]: ATI Technologies Inc IXP SB400 PCI-ISA Bridge [1002:4377] (rev 80)
00:14.4 PCI bridge [0604]: ATI Technologies Inc IXP SB400 PCI-PCI Bridge [1002:4371] (rev 80)
00:18.0 Host bridge [0600]: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration [1022:1100]
00:18.1 Host bridge [0600]: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map [1022:1101]
00:18.2 Host bridge [0600]: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller [1022:1102]
00:18.3 Host bridge [0600]: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control [1022:1103]
01:05.0 VGA compatible controller [0300]: ATI Technologies Inc RS485 [Radeon Xpress 1100 IGP] [1002:5975]
08:01.0 CardBus bridge [0607]: ENE Technology Inc CB-712/4 Cardbus Controller [1524:1412] (rev 10)
08:01.1 FLASH memory [0501]: ENE Technology Inc ENE PCI Memory Stick Card Reader Controller [1524:0530] (rev 01)
08:01.2 SD Host controller [0805]: ENE Technology Inc ENE PCI Secure Digital Card Reader Controller [1524:0550] (rev 01)
08:01.3 FLASH memory [0501]: ENE Technology Inc FLASH memory: ENE Technology Inc: [1524:0520] (rev 01)
08:01.4 FLASH memory [0501]: ENE Technology Inc SD/MMC Card Reader Controller [1524:0551] (rev 01)
08:02.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ [10ec:8139] (rev 10)
08:04.0 Ethernet controller [0200]: Atheros Communications Inc. AR2413 802.11bg NIC [168c:001a] (rev 01)

Actually the latest 2.6.26.3-27.fc9 works fine. I isolated the "fixing" patch and applied it to 2.6.26.3-17.fc9 and 2.6.27-0.290.rc5.fc10. Now both load fine(I got the udev hanging with the patch https://bugzilla.redhat.com/show_bug.cgi?id=458901). Can you confirm this for you?

Comment 31 Joshua Covington 2008-09-02 00:31:38 UTC
Created attachment 315500 [details]
linux-2.6-x86-32-amd-c1e-force-timer-broadcast-late.patch

This is tha patch. It is working for me.

Comment 32 Tom London 2008-09-02 00:48:58 UTC
Uhh, I don't have AMD.  

Any reason to believe this would work on Intel Core Duo?

Comment 33 Joshua Covington 2008-09-02 01:01:44 UTC
(In reply to comment #32)
> Uhh, I don't have AMD.  
> 
> Any reason to believe this would work on Intel Core Duo?

I know thinkpad t6os come with intel but in comment 28 I see that Gideon Mayhak has a host bridge from AMD. Therefore I thought it could help. Can you give it a try anyway?

the patch concerns arch/x86/kernel/apic_32.c and it is not amd specific (at least from what i see in the code). try it.

Comment 34 Tom London 2008-09-02 13:24:34 UTC
I set this up last night to build a patched 0.295, but two things:

1. I think (unpatched) 0.295 has booted perfectly every time.  Is it possible that something else changed (for Intel)?

2. I forgot about the firmware and headers packages. What is the magic rpmbuild incantation to build these too?

Comment 35 Joshua Covington 2008-09-02 15:40:25 UTC
1. if unpatched 0.295 works fine then this has already been fixed. in my case nothing from 0.rc2 can boot although it is supposed to have another c1e logic in the 2.6.27 series.

2. you don't need them. in the kernel.spec file set only with_up and base_only to 1, everything else to 0. this way you'll ge the basic kernel (with -bb --target=i686). it is enough for testing. to even speed things up i install the default i686 kernel, then only rebuild the bzImage and replace the original one. It takes about 5 mins on my turoin x2 64 for rebuilding.

3. for headers and firmware packges just set the corrsponding line in the kernel.spec to 1. But you can download them from koji, precompiled.

Comment 36 Joshua Covington 2008-09-02 15:42:16 UTC
is the patch working for you, or you don't need it?

Comment 37 Tom London 2008-09-02 16:06:56 UTC
I haven't installed rpm with patch.  0.295 hasn't failed yet.

I build with 'buildid' set so I can quickly figure out with kernel is which.

I'll figure out how to build the other packages in case 0.295 starts failing.

Comment 38 Tom London 2008-09-04 13:43:56 UTC
This problem has "gone away" with kernels 0.295 and later.

Cannot see from the changelog what was 'fixed', but I'm happy nonetheless.

Can someone verify this on kernels 0.295 or later?

Comment 39 Yanko Kaneti 2008-09-04 13:54:31 UTC
I still get the occasional hang with 2.6.27-0.303.rc5.git5
No boot with modeprobedebug has failed so far. Annoying indeed.
This on a ASUS mobo (PVE-VM DO) and Dual core Celeron E1200

Comment 40 Ralf Ertzinger 2008-09-10 13:38:23 UTC
314 still hangs on my X60s

Comment 41 Tom London 2008-09-10 13:44:06 UTC
After "going away" for builds >=0.295 and < 0.314, this has "come back" for me with 0.314 and 0.317 on my Thinkpad X60 (Intel 945/i810 graphics).

Whatever is at issue (modesetting?) must still not be perfect.....

Comment 42 Tom London 2008-09-10 17:18:04 UTC
Issue continue with 0.321.

I got 4 'udev starting' freezes in a row.....

Comment 43 Tom London 2008-09-17 14:11:37 UTC
Issue continues with 0.329.i686: froze at 'Starting udev' 2 of 4 boots (on Thinkpad X60).

Comment 44 Tom London 2008-09-23 15:45:48 UTC
Continue to have this problem with kernel-2.6.27-0.347.rc7.git1.fc10.i686.

4 straight boot freezes in a row before one finally booted with 'modprobedebug'

Thinkpad X60, Intel graphics.

Comment 45 Tom London 2008-09-25 16:30:31 UTC
Last few boots, I've set 'modprobedebug' on boot.

When it freezes, the last printed line reposts loading iwl3945.ko.

When it "boots", the next printed line reports loading bluetooth.ko.

Not sure this is pertinent......

Comment 46 Tom London 2008-10-03 14:05:10 UTC
Continue to have problem described above with kernel-2.6.27-0.382.rc8.git4.fc10.i686 on Thinkpad X60.

I'm testing also on new x86-64 install on Thinkpad X61 and haven't seen this problem yet.

Comment 47 Tom London 2008-10-06 16:00:00 UTC
Problem persists with 0.393.fc10.i686

Comment 48 Tom London 2008-10-10 16:45:03 UTC
Not sure how relevant it is, but since updating to the "newer, faster" MAKEDEV, every boot has just worked.

Something in the timing?

Anyway, I continue to test... (by booting frequently).