Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1688968
Summary: | after upgrade rawhide "Virtual network 'default': NAT" became Inactive | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Mikhail <mikhail.v.gavrilov> | ||||||
Component: | libvirt | Assignee: | Libvirt Maintainers <libvirt-maint> | ||||||
Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||
Severity: | unspecified | Docs Contact: | |||||||
Priority: | unspecified | ||||||||
Version: | 30 | CC: | agedosier, berrange, clalancette, crobinso, itamar, jforbes, jpokorny, laine, libvirt-maint, paul.destefano-redhat2, veillard, virt-maint | ||||||
Target Milestone: | --- | ||||||||
Target Release: | --- | ||||||||
Hardware: | Unspecified | ||||||||
OS: | Unspecified | ||||||||
Whiteboard: | |||||||||
Fixed In Version: | libvirt-5.1.0-3.fc30 | Doc Type: | If docs needed, set a value | ||||||
Doc Text: | Story Points: | --- | |||||||
Clone Of: | Environment: | ||||||||
Last Closed: | 2019-03-29 19:19:07 UTC | Type: | Bug | ||||||
Regression: | --- | Mount Type: | --- | ||||||
Documentation: | --- | CRM: | |||||||
Verified Versions: | Category: | --- | |||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||
Embargoed: | |||||||||
Attachments: |
|
Description
Mikhail
2019-03-14 19:49:35 UTC
Last working libvirt: # rpm -qa | grep libvirt | sort libvirt-daemon-5.0.0-2.fc30.x86_64 libvirt-daemon-driver-interface-5.0.0-2.fc30.x86_64 libvirt-daemon-driver-network-5.0.0-2.fc30.x86_64 libvirt-daemon-driver-nodedev-5.0.0-2.fc30.x86_64 libvirt-daemon-driver-nwfilter-5.0.0-2.fc30.x86_64 libvirt-daemon-driver-qemu-5.0.0-2.fc30.x86_64 libvirt-daemon-driver-secret-5.0.0-2.fc30.x86_64 libvirt-daemon-driver-storage-5.0.0-2.fc30.x86_64 libvirt-daemon-driver-storage-core-5.0.0-2.fc30.x86_64 libvirt-daemon-driver-storage-disk-5.0.0-2.fc30.x86_64 libvirt-daemon-driver-storage-gluster-5.0.0-2.fc30.x86_64 libvirt-daemon-driver-storage-iscsi-5.0.0-2.fc30.x86_64 libvirt-daemon-driver-storage-iscsi-direct-5.0.0-2.fc30.x86_64 libvirt-daemon-driver-storage-logical-5.0.0-2.fc30.x86_64 libvirt-daemon-driver-storage-mpath-5.0.0-2.fc30.x86_64 libvirt-daemon-driver-storage-rbd-5.0.0-2.fc30.x86_64 libvirt-daemon-driver-storage-scsi-5.0.0-2.fc30.x86_64 libvirt-daemon-driver-storage-sheepdog-5.0.0-2.fc30.x86_64 libvirt-daemon-driver-storage-zfs-5.0.0-2.fc30.x86_64 libvirt-daemon-kvm-5.0.0-2.fc30.x86_64 libvirt-gconfig-2.0.0-3.fc30.x86_64 libvirt-glib-2.0.0-3.fc30.x86_64 libvirt-gobject-2.0.0-3.fc30.x86_64 libvirt-libs-5.0.0-2.fc30.x86_64 python3-libvirt-5.1.0-1.fc31.x86_64 Libvirt creates the reference iptables rule at startup. If it is missing then most likely one or more of the required iptables modules is missing. Usually this is seen on machines where the admin has blacklisted ipv6. The current libvirt code requires the ip6tables modules as well as iptables modules, though we're likely to relax that requirement given how many people have hit this. > Usually this is seen on machines where the admin has blacklisted ipv6.
I did not blacklisted ipv6 on my machine.
I just updated libvirt from 5.0.0-2 to 5.1.0-2.
Interesting, can you collect some debug info Edit /etc/libvirt/libvirtd.conf and set log_filters="1:firewall 1:network 1:iptables" log_outputs="1:file:/var/log/libvirt/libvirtd.log" The systemctl start libvirtd And attempt to start the network again Then attach that logfile to this bugzilla. It would also be useful to get attachment with output from iptables-save ip6tables-save # journalctl -b -u libvirtd --no-hostname > Mar 15 14:15:16 systemd[1]: Starting Virtualization daemon... > Mar 15 14:15:17 systemd[1]: Started Virtualization daemon. > Mar 15 14:15:19 libvirtd[4699]: libvirt version: 5.0.0, package: 2.fc30 (Fedora Project, 2019-02-01-18:25:08, ) > [...] > Mar 15 14:15:19 libvirtd[4699]: Unable to open /dev/net/tun, is tun module loaded?: No such file or directory > Mar 15 14:15:19 libvirtd[4699]: Unable to open /dev/net/tun, is tun module loaded?: No such file or directory > Mar 15 14:15:19 libvirtd[4699]: Unable to open /dev/net/tun, is tun module loaded?: No such file or directory > Mar 15 14:15:19 libvirtd[4699]: Unable to open /dev/net/tun, is tun module loaded?: No such file or directory # lsmod | grep tun > [nothing] # modprobe tun # echo $? > 0 # lsmod | grep tun > tun 61440 0 # systemctl restart libvirtd --> everything works now, "tun" kernel module needs to be loaded explicitly now, it seems # uname -r > 5.1.0-0.rc0.git7.1.fc31.x86_64 That's a bit odd - I'm not sure why missing tun module would cause libvirtd to fail to setup firewall rules Created attachment 1544499 [details]
libvirtd.log
Created attachment 1544501 [details]
iptables-save and ip6tables-save
# ifconfig enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.1.68 netmask 255.255.255.0 broadcast 192.168.1.255 inet6 fe80::f9a9:2a39:2f99:e003 prefixlen 64 scopeid 0x20<link> ether 4c:ed:fb:75:5b:ab txqueuelen 1000 (Ethernet) RX packets 2936085 bytes 3785903251 (3.5 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 880090 bytes 123691587 (117.9 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device memory 0xfe400000-fe41ffff lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 1000 (Local Loopback) RX packets 119 bytes 6385 (6.2 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 119 bytes 6385 (6.2 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 vpn0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST> mtu 1406 inet 10.76.146.109 netmask 255.255.254.0 destination 10.76.146.109 inet6 fe80::7230:2390:76f4:7ce2 prefixlen 64 scopeid 0x20<link> unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 txqueuelen 500 (UNSPEC) RX packets 77325 bytes 22338225 (21.3 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 99078 bytes 17852270 (17.0 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 Thanks, so that shows two things - there are no ip6tables top level chains at all - firewalld is reporting that it doens't have ip6tables support 2019-03-15 15:13:03.859+0000: 31773: info : virFirewallApplyRule:723 : Applying rule '/usr/sbin/ip6tables --table filter --list-rules' 2019-03-15 15:13:03.862+0000: 31773: error : virFirewallDApplyRule:332 : COMMAND_FAILED: UNKNOWN_ERROR: 'ip6tables' backend does not exist You say you didn't disable ipv6, but it none the less looks like something related to ipv6 is disabled. It could be just general Fedora rawhide explodyness > You say you didn't disable ipv6, but it none the less looks like something related to ipv6 is disabled.
Yes I didn't disable ipv6, but my ISP not support ipv6 and didn't gives ipv6 address to my computer.
I think it could well be something that's changed in firewalld. eg I think they might have switched over to the NFT kernel backend and be relying on compat support for iptables/ip6tables. I'm installing a new rawhide VM to debug this further myself. re [comment 5]: I thought I was upon something that's a universal solution for the problem at hand, since I also originally suffered the problem with available networks marked as "inactive". Perhaps the surface is more convoluted, and I was just lucky to help myself with said easy step. $ rpm -q firewalld nftables iptables systemd > firewalld-0.6.3-2.fc30.noarch > nftables-0.9.0-4.fc30.x86_64 > iptables-1.8.0-5.fc30.x86_64 > systemd-241-2.gita09c170.fc31.x86_64 So far I can reproduce the problem with missing /dev/net/tun device & tun kmod. This appears to be a regression in Fedora 31 - not sure if is intentional or not (https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/RW4WZYLPIMZ6K4JP5VYIPOWXP7LIFV3I/) I cannot reproduce the missing IPv6 support though. A plain install of F31 gets a link-local IPv6 address regardless of whether the upstream link supports it or not. None the less it is possible for people to blacklist ipv6.ko so we'll need to cope with that better. The following series is merged upstream & will need backporting to Fedora 30 https://www.redhat.com/archives/libvir-list/2019-March/msg01218.html WHile doing that we should also grab this other network related fix https://www.redhat.com/archives/libvir-list/2019-March/msg00881.html libvirt-5.1.0-3.fc30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-f3e6d3e0a5 libvirt-5.1.0-3.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-f3e6d3e0a5 libvirt-5.1.0-3.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report. |