Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1708717

Summary: backport 5.x IPv6 fix: neighbour: arp_cache: neighbor table overflow!
Product: [Fedora] Fedora Reporter: Jan Kratochvil <jan.kratochvil>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 29CC: airlied, bskeggs, edgar.hoch, hdegoede, ichavero, itamar, jan.kratochvil, jarodwilson, jeremy, jglisse, john.j5live, jonathan, josef, kas, kernel-maint, linville, mchehab, mjg59, steved
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: kernel-5.1.8-300.fc30 kernel-5.1.8-200.fc29 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-12 14:47:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
kernel.spec patch to build F-29 kernel none

Description Jan Kratochvil 2019-05-10 16:39:38 UTC 
1. Please describe the problem:

After few days of running kernel-5.0.x my IPv6 connectivity gets very unreliable and kernel log is full of:
  neighbour: arp_cache: neighbor table overflow

2. What is the Version-Release number of the kernel:

kernel-5.0.6-200.fc29.x86_64 fails
kernel-5.0.11-200.fc29.x86_64 fails

3. Did it work previously in Fedora? If so, what kernel version did the issue
   *first* appear?  Old kernels are available for download at
   https://koji.fedoraproject.org/koji/packageinfo?packageID=8 :

kernel-4.20.16-200.fc29.x86_64 works fine

4. Can you reproduce this issue? If so, please provide the steps to reproduce
   the issue below:

Yes, run IPv6 on local gigabit ethernet over openvpn and ping it, sometimes it stops responding:
17:24:08.288597 IP6 fc00::67:2 > fc00::67:1: ICMP6, echo request, seq 717, length 1408
17:24:08.288632 IP6 fc00::67:1 > fc00::67:2: ICMP6, echo reply, seq 717, length 1408
17:24:08.298252 IP6 fc00::67:2 > fc00::67:1: ICMP6, echo request, seq 718, length 1408
17:24:08.298287 IP6 fc00::67:1 > fc00::67:2: ICMP6, echo reply, seq 718, length 1408
17:24:08.308274 IP6 fc00::67:2 > fc00::67:1: ICMP6, echo request, seq 719, length 1408
17:24:08.319391 IP6 fc00::67:2 > fc00::67:1: ICMP6, echo request, seq 720, length 1408


5. Does this problem occur with the latest Rawhide kernel? To install the
   Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by
   ``sudo dnf update --enablerepo=rawhide kernel``:

Latest Rawhide is kernel-5.1.0-1.fc31 and it still does not contain the fix so I expect it will still fail but I have not verified that.

6. Are you running any modules that not shipped with directly Fedora's kernel?:

No.

7. Please attach the kernel logs. You can get the complete kernel log
   for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the
   issue occurred on a previous boot, use the journalctl ``-b`` flag.

May 10 17:42:04 host1 kernel: neighbour: ndisc_cache: neighbor table overflow!
May 10 17:42:04 host1 kernel: neighbour: ndisc_cache: neighbor table overflow!
May 10 17:42:04 host1 kernel: neighbour: ndisc_cache: neighbor table overflow!
May 10 17:42:04 host1 named[3391702]: ../../../../lib/isc/unix/socket.c:2173: unexpected error:
May 10 17:42:04 host1 named[3391702]: internal_send: 2001:500:c::1#53: Invalid argument
May 10 17:42:04 host1 kernel: net_ratelimit: 14 callbacks suppressed
May 10 17:42:04 host1 kernel: neighbour: ndisc_cache: neighbor table overflow!

Fix is:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=64c6f4bbca748c3b2101469a76d88b7cd1c00476
Comment 1 Edgar Hoch 2019-05-18 18:40:10 UTC 
I get a similar error running Fedora 29:

kernel-5.0.16-200.fc29.x86_64
bind-chroot-9.11.6-2.P1.fc29.x86_64
dhcp-server-4.3.6-31.fc29.x86_64

MAC and IP addresses are replaced by XXX and YYY:

Mai 18 20:34:16 kernel: neighbour: arp_cache: neighbor table overflow!
Mai 18 20:34:16 kernel: neighbour: arp_cache: neighbor table overflow!
Mai 18 20:34:16 kernel: neighbour: arp_cache: neighbor table overflow!
Mai 18 20:34:16 named[1367]: ../../../../lib/isc/unix/socket.c:2176: unexpected error:
Mai 18 20:34:16 kernel: neighbour: arp_cache: neighbor table overflow!
Mai 18 20:34:16 named[1367]: internal_send: YYY#59267: Invalid argument
Mai 18 20:34:16 named[1367]: client @0xXXX YYY#59267 (fedoraproject.org): view ims: error sending response: invalid file
Mai 18 20:34:16 named[1367]: ../../../../lib/isc/unix/socket.c:2176: unexpected error:
Mai 18 20:34:16 named[1367]: internal_send: YYY#43658: Invalid argument

It seams that the patch linked in previous comment is still not in released kernel?

Is there a workaround for this problem (beside restarting the server very often)?
Comment 2 Jan Kratochvil 2019-05-18 19:07:47 UTC 
Created attachment 1570674 [details]
kernel.spec patch to build F-29 kernel

(In reply to Edgar Hoch from comment #1)
> It seams that the patch linked in previous comment is still not in released
> kernel?

It is present in (not in 5.1.x):
kernel-5.2.0-0.rc0.git6.1.fc31 = cbd87613cc306a7ccd5ce4006daf8dc737922c3f = has IPv6

But fc31 kernels are not installable into F-29 OS.  They are installable into F-30 OS (but upgrade F-29 to F-30 failed for me due to LUKS today).


> Is there a workaround for this problem (beside restarting the server very
> often)?

I build my own F-29 kernels with this attached patch.
Comment 3 Jan Kratochvil 2019-05-28 21:48:17 UTC 
The problem still affects also: kernel-5.0.19-200.fc29.x86_64
Comment 4 Jan "Yenya" Kasprzak 2019-06-05 17:37:24 UTC 
Also affected:  5.1.5-300.fc30.x86_64

On my server/router the problem manifests 3 to 4 days after reboot.
Comment 5 Jan "Yenya" Kasprzak 2019-06-10 08:58:14 UTC 
FWIW, also affected: 5.1.6-300.fc30.x86_64
Comment 6 Jan Kratochvil 2019-06-10 09:05:46 UTC 
And also affected kernel-5.1.7-200.fc29.x86_64 :-)
Comment 7 Fedora Update System 2019-06-10 15:19:53 UTC 
FEDORA-2019-c03eda3cc6 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-c03eda3cc6
Comment 8 Fedora Update System 2019-06-10 15:19:54 UTC 
FEDORA-2019-83858fc57b has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-83858fc57b
Comment 9 Fedora Update System 2019-06-11 01:19:17 UTC 
kernel-5.1.8-300.fc30, kernel-headers-5.1.8-300.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-c03eda3cc6
Comment 10 Fedora Update System 2019-06-11 01:45:36 UTC 
kernel-5.1.8-200.fc29, kernel-headers-5.1.8-200.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-83858fc57b
Comment 11 Fedora Update System 2019-06-12 14:47:57 UTC 
kernel-5.1.8-300.fc30, kernel-headers-5.1.8-300.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.
Comment 12 Fedora Update System 2019-06-13 01:38:25 UTC 
kernel-5.1.8-200.fc29, kernel-headers-5.1.8-200.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.