Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1592932
Summary: | no network access in containers when doing 'podman run' on RHELAH | |||
---|---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | Micah Abbott <miabbott> | |
Component: | podman | Assignee: | Dan Williams <dcbw> | |
Status: | CLOSED CURRENTRELEASE | QA Contact: | Martin Jenner <mjenner> | |
Severity: | high | Docs Contact: | ||
Priority: | high | |||
Version: | 7.5 | CC: | arusso, atomic-bugs, dwalsh, fkluknav, lsm5, mheon, umohnani | |
Target Milestone: | rc | Keywords: | Extras | |
Target Release: | --- | |||
Hardware: | Unspecified | |||
OS: | Unspecified | |||
Whiteboard: | ||||
Fixed In Version: | Doc Type: | If docs needed, set a value | ||
Doc Text: | Story Points: | --- | ||
Clone Of: | ||||
: | 1593419 (view as bug list) | Environment: | ||
Last Closed: | 2019-03-01 15:43:42 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | ||||
Bug Blocks: | 1593419 |
Description
Micah Abbott
2018-06-19 15:33:57 UTC
> On a RHELAH 7.5.1-1 system, I'm able to get network access in a container when using `podman run`.
That should say "I'm *unable* to get network access" obviously
podman version Micah, have you tried to build podman from github and see if it works? Dan, I did build from git master; see below RPM version -------------- # podman version Version: 0.4.1 Go Version: go1.9.2 OS/Arch: linux/amd64 git master version --------------------- # /srv/podman version Version: 0.6.4-dev Go Version: go1.10.3 OS/Arch: linux/amd64 # date; /srv/podman run -it docker.io/alpine ping -c 5 1.1.1.1; date Tue Jun 19 17:02:13 UTC 2018 PING 1.1.1.1 (1.1.1.1): 56 data bytes --- 1.1.1.1 ping statistics --- 5 packets transmitted, 0 packets received, 100% packet loss Tue Jun 19 17:02:27 UTC 2018 Surprisingly, I'm able to get network access on regular RHEL Server: # cat /etc/os-release NAME="Red Hat Enterprise Linux Server" VERSION="7.5 (Maipo)" ID="rhel" ID_LIKE="fedora" VARIANT="Server" VARIANT_ID="server" VERSION_ID="7.5" PRETTY_NAME="Red Hat Enterprise Linux Server 7.5 (Maipo)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:7.5:GA:server" HOME_URL="https://www.redhat.com/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" REDHAT_BUGZILLA_PRODUCT_VERSION=7.5 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="7.5" # rpm -q containernetworking-plugins podman runc containernetworking-plugins-0.7.0-4.gitb51d327.el7.x86_64 podman-0.4.1-4.gitb51d327.el7.x86_64 runc-1.0.0-27.rc5.dev.git4bb1fe4.el7.x86_64 # podman version Version: 0.4.1 Go Version: go1.9.2 OS/Arch: linux/amd64 # date; podman run -it docker.io/alpine ping -c 5 1.1.1.1; date Tue Jun 19 13:04:59 EDT 2018 PING 1.1.1.1 (1.1.1.1): 56 data bytes 64 bytes from 1.1.1.1: seq=0 ttl=48 time=11.128 ms 64 bytes from 1.1.1.1: seq=1 ttl=48 time=11.122 ms 64 bytes from 1.1.1.1: seq=2 ttl=48 time=11.197 ms 64 bytes from 1.1.1.1: seq=3 ttl=48 time=11.114 ms 64 bytes from 1.1.1.1: seq=4 ttl=48 time=11.078 ms --- 1.1.1.1 ping statistics --- 5 packets transmitted, 5 packets received, 0% packet loss round-trip min/avg/max = 11.078/11.127/11.197 ms Tue Jun 19 13:05:03 EDT 2018 Dan suggested there might be an iptables/CNI problem, so I looked at the current state of the iptables rules on both hosts. There are definitely differences, but I'm unsure if there is a smoking gun. RHELAH 7.5.1-1 ---------------- # iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy DROP) target prot opt source destination ACCEPT all -- 10.88.0.7 anywhere DOCKER-ISOLATION all -- anywhere anywhere DOCKER all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain DOCKER (1 references) target prot opt source destination Chain DOCKER-ISOLATION (1 references) target prot opt source destination RETURN all -- anywhere anywhere RHEL 7 Server -------------- # iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED ACCEPT all -- anywhere anywhere INPUT_direct all -- anywhere anywhere INPUT_ZONES_SOURCE all -- anywhere anywhere INPUT_ZONES all -- anywhere anywhere DROP all -- anywhere anywhere ctstate INVALID REJECT all -- anywhere anywhere reject-with icmp-host-prohibited Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED ACCEPT all -- anywhere anywhere FORWARD_direct all -- anywhere anywhere FORWARD_IN_ZONES_SOURCE all -- anywhere anywhere FORWARD_IN_ZONES all -- anywhere anywhere FORWARD_OUT_ZONES_SOURCE all -- anywhere anywhere FORWARD_OUT_ZONES all -- anywhere anywhere DROP all -- anywhere anywhere ctstate INVALID REJECT all -- anywhere anywhere reject-with icmp-host-prohibited Chain OUTPUT (policy ACCEPT) target prot opt source destination OUTPUT_direct all -- anywhere anywhere Chain FORWARD_IN_ZONES (1 references) target prot opt source destination FWDI_public all -- anywhere anywhere [goto] FWDI_public all -- anywhere anywhere [goto] Chain FORWARD_IN_ZONES_SOURCE (1 references) target prot opt source destination Chain FORWARD_OUT_ZONES (1 references) target prot opt source destination FWDO_public all -- anywhere anywhere [goto] FWDO_public all -- anywhere anywhere [goto] Chain FORWARD_OUT_ZONES_SOURCE (1 references) target prot opt source destination Chain FORWARD_direct (1 references) target prot opt source destination Chain FWDI_public (2 references) target prot opt source destination FWDI_public_log all -- anywhere anywhere FWDI_public_deny all -- anywhere anywhere FWDI_public_allow all -- anywhere anywhere ACCEPT icmp -- anywhere anywhere Chain FWDI_public_allow (1 references) target prot opt source destination Chain FWDI_public_deny (1 references) target prot opt source destination Chain FWDI_public_log (1 references) target prot opt source destination Chain FWDO_public (2 references) target prot opt source destination FWDO_public_log all -- anywhere anywhere FWDO_public_deny all -- anywhere anywhere FWDO_public_allow all -- anywhere anywhere Chain FWDO_public_allow (1 references) target prot opt source destination Chain FWDO_public_deny (1 references) target prot opt source destination Chain FWDO_public_log (1 references) target prot opt source destination Chain INPUT_ZONES (1 references) target prot opt source destination IN_public all -- anywhere anywhere [goto] IN_public all -- anywhere anywhere [goto] Chain INPUT_ZONES_SOURCE (1 references) target prot opt source destination Chain INPUT_direct (1 references) target prot opt source destination Chain IN_public (2 references) target prot opt source destination IN_public_log all -- anywhere anywhere IN_public_deny all -- anywhere anywhere IN_public_allow all -- anywhere anywhere ACCEPT icmp -- anywhere anywhere Chain IN_public_allow (1 references) target prot opt source destination ACCEPT tcp -- anywhere anywhere tcp dpt:ssh ctstate NEW Chain IN_public_deny (1 references) target prot opt source destination Chain IN_public_log (1 references) target prot opt source destination Chain OUTPUT_direct (1 references) target prot opt source destination Sadly, this is also affecting RHELAH 7.5.2 # rpm-ostree status State: idle; auto updates disabled Deployments: ● ostree://custom:rhel-atomic-host/7/x86_64/standard Version: 7.5.2 (2018-06-09 02:40:55) Commit: db4a302e874cdd9cc9517a63133cfdf05e23cb684faae166b444c74cf7c146e8 GPGSignature: Valid signature by 567E347AD0044ADE55BA8A5F199E2F91FD431D51 ostree://rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard Version: 7.5.1 (2018-05-08 16:36:53) Commit: c0211e0b703930dd0f0df8b9f5e731901fce8e15e00b3bc76d3cf00df44eb6e8 GPGSignature: Valid signature by 567E347AD0044ADE55BA8A5F199E2F91FD431D51 # rpm -q containernetworking-plugins podman runc containernetworking-plugins-0.7.0-101.el7.x86_64 podman-0.6.1-3.git3e0ff12.el7.x86_64 runc-1.0.0-27.rc5.dev.git4bb1fe4.el7.x86_64 # podman version Version: 0.6.1 Go Version: go1.9.2 OS/Arch: linux/amd64 # podman run docker.io/alpine ping -c 5 1.1.1.1 PING 1.1.1.1 (1.1.1.1): 56 data bytes --- 1.1.1.1 ping statistics --- 5 packets transmitted, 0 packets received, 100% packet loss Additional workaround if you don't want to use `--net=host` <baude> dcbw, if a guy was stuck with a binary that didn't have that, is there a simple iptables command he could run ? <dcbw> baude: if you have the container's IP address you can: <dcbw> iptables -t nat -A FORWARD -d <ipaddr> -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT <baude> k <dcbw> or for the entire bridge, iptables -t nat -A FORWARD -o <cni bridge name> -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT @dcbw suggested that this PR would address this problem: https://github.com/containernetworking/plugins/pull/75 Removing the request for blocker: - this isn't technically a regression - there have been no customer cases about this - there is a workaround available Seems that PR is still languishing. Latest extras-rhel-7.6 branch in dist-git has the podman firewall workaround stuff in version 1.0.1. I believe this bug is fixed because of that. |