Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1628209

Summary: kernel 4.18.5-200.fc28.armv7hl has non-functional ethernet on wandboard quad
Product: [Fedora] Fedora Reporter: Steven A. Falco <stevenfalco>
Component: kernelAssignee: Peter Robinson <pbrobinson>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 28CC: airlied, bskeggs, ewk, fzatlouk, hdegoede, ichavero, itamar, jarodwilson, jglisse, john.j5live, jonathan, josef, kernel-maint, linville, mchehab, mjg59, pbrobinson, pwhalen, steved, tkrantz
Target Milestone: ---   
Target Release: ---   
Hardware: armv7hl   
OS: Linux   
Whiteboard: AcceptedFreezeException
Fixed In Version: kernel-4.18.16-300.fc29 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-10-23 15:03:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 245418, 1517014    

Description Steven A. Falco 2018-09-12 13:09:02 UTC
Description of problem:

I just upgraded my wandboard quad.  This brought in kernel 4.18.5-200.fc28.armv7hl.

The wandboard booted, but failed to mount several NFS drives during boot.  After I logged in, I was unable to ping any machines on the LAN.  The ethernet interface appears to be non-functional.

I rebooted the wandboard and selected previous kernel 4.17.19-200.fc28.armv7hl and the ethernet worked properly.

Thus, it appears that kernel 4.18.5-200.fc28.armv7hl breaks ethernet on my wandboard quad.


Version-Release number of selected component (if applicable):

4.18.5-200.fc28.armv7hl


How reproducible:

100%


Steps to Reproduce:
1. boot wandboard quad with kernel 4.18.5-200.fc28.armv7hl
2.
3.

Actual results:

Ethernet does not work, as described above


Expected results:

Ethernet works properly

Additional info:

Comment 1 Steven A. Falco 2018-10-01 12:59:18 UTC
Tested kernel 4.18.10-200.  Unfortunately ethernet is still broken.

Comment 2 Steven A. Falco 2018-10-15 12:57:45 UTC
Tested kernel 4.18.13-200.  Ethernet is still broken.

Comment 3 Paul Whalen 2018-10-15 13:59:30 UTC
Last working kernel was 4.18.0-0.rc0.git2.1.fc29.

Comment 4 Tim Krantz 2018-10-15 18:42:57 UTC
I have this same issue with my wandboard quad.  The only information I can add is that if I stop the boot process and get a uboot prompt then try to boot from the network Uboot is able to obtain an IP address.  The kernel is not able to do this.

Comment 5 Peter Robinson 2018-10-19 10:51:49 UTC
I can reproduce, the issue is now to work out what caused the regression.

Comment 6 Peter Robinson 2018-10-19 18:43:52 UTC
So there's a couple of things coming into play here. There was a bug in the 4.18 cycle that caused boot issues with ARMv7 due to BPF JIT which causes an issue on the device, but when that kernel is built with BPF JIT disabled networking works fine. So for this actual problem the rc0.git[3-5].1 kernels are fine.

Just building/testing kernels with the bpf jit disabled to work out which is the first kernel where this actually stopped working so I can then bisect it from there.

Comment 7 Peter Robinson 2018-10-19 18:45:06 UTC
Interestingly also a i.MX^sx device works just fine

Comment 8 Peter Robinson 2018-10-20 15:40:23 UTC
(In reply to Peter Robinson from comment #7)
> Interestingly also a i.MX^sx device works just fine

i.MX6sx even.

So the regression happened first with the 4.18.0-0.rc0.git10.1.fc29 kernel.

Comment 9 Peter Robinson 2018-10-20 15:41:33 UTC
Which for reference is somewhere in the upstream commit range of 2837461dbe6f4a9acc0d86f88825888109211c99..4c5e8fc62d6a63065eeae80808c498d1dcfea4f4

Comment 10 Peter Robinson 2018-10-21 15:56:21 UTC
This is fixed in the 4.18.16 kernel. Please test.

For reference, in 4.18 the OF_MDIO/MDIO_DEVICE/FIXED_PHY/PHYLIB driver components were fixed to build modular so we moved them from built in to modular. It seems the micrel PHY used in some imx6 devices works fine when using the above bits built modular but the at803x used by Hummingboard/Wandboard and friends doesn't. I've reverted to building it back into the kernel until I can engage with upstream to get it fixed.

Comment 11 Steven A. Falco 2018-10-21 16:24:08 UTC
Confirmed.  Kernel 4.18.16 fixes Ethernet bug 1628209 on my Wandboard Quad.

Thanks for the fix!

Comment 12 Fedora Blocker Bugs Application 2018-10-22 07:30:15 UTC
Proposed as a Freeze Exception for 29-final by Fedora user pbrobinson using the blocker tracking app because:

 imx6 is a key ARMv7 SoC with numerous devices, including widely used Wandboard and Cubox-i

Comment 13 Fedora Update System 2018-10-22 08:25:37 UTC
kernel-headers-4.18.16-300.fc29 kernel-4.18.16-300.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2018-dbea673f8a

Comment 14 Fedora Update System 2018-10-22 08:27:36 UTC
kernel-headers-4.18.16-200.fc28 kernel-4.18.16-200.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-2a2a5be13c

Comment 15 Fedora Update System 2018-10-22 08:33:06 UTC
kernel-headers-4.18.16-100.fc27 kernel-4.18.16-100.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-edf9c345c3

Comment 16 FrantiĊĦek Zatloukal 2018-10-22 16:47:23 UTC
Discussed during the 2018-10-22 blocker review meeting: [1]

The decision to classify this bug as an AcceptedFreezeException was made:

"This bug has been noted to be a "nice-to-have" fix, so we accept it as an AcceptedFreezeException."

[1] https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2018-10-22/f29-blocker-review.2018-10-22-16.00.log.txt

Comment 17 Fedora Update System 2018-10-22 19:23:23 UTC
kernel-4.18.16-300.fc29, kernel-headers-4.18.16-300.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-dbea673f8a

Comment 18 Fedora Update System 2018-10-22 23:22:25 UTC
kernel-4.18.16-100.fc27, kernel-headers-4.18.16-100.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-edf9c345c3

Comment 19 Fedora Update System 2018-10-22 23:54:26 UTC
kernel-4.18.16-200.fc28, kernel-headers-4.18.16-200.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-2a2a5be13c

Comment 20 Fedora Update System 2018-10-23 15:03:39 UTC
kernel-4.18.16-300.fc29, kernel-headers-4.18.16-300.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.

Comment 21 Fedora Update System 2018-10-23 21:08:39 UTC
kernel-4.18.16-200.fc28, kernel-headers-4.18.16-200.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.

Comment 22 Fedora Update System 2018-10-25 22:00:38 UTC
kernel-4.18.16-100.fc27, kernel-headers-4.18.16-100.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.