Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1584909
Summary: | oc cluster up does not work on docker-2:1.13.1-56.git6c336e4.fc28.x86_64 | ||
---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Jason Montleon <jmontleo> |
Component: | docker | Assignee: | Daniel Walsh <dwalsh> |
Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
Severity: | unspecified | Docs Contact: | |
Priority: | unspecified | ||
Version: | 28 | CC: | adimania, admiller, amurdaca, dustymabe, dwalsh, filbranden, fkluknav, ichavero, jcajka, jpazdziora, jwhiting, lsm5, marianne, nalin, rh-bugzilla, santiago, tom81094, tomek, ttomecek, twaugh, vbatts |
Target Milestone: | --- | ||
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | docker-1.13.1-59.gitaf6b32b.fc28 docker-1.13.1-59.gitaf6b32b.fc27 | Doc Type: | If docs needed, set a value |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2018-06-13 15:18:25 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Jason Montleon
2018-05-31 22:23:54 UTC
The bug usually starts at the point origin needs to pull down the openshift/origin-web-console image: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 1m default-scheduler Successfully assigned webconsole-7dfbffd44d-bz44s to localhost Normal SuccessfulMountVolume 1m kubelet, localhost MountVolume.SetUp succeeded for volume "webconsole-config" Normal SuccessfulMountVolume 1m kubelet, localhost MountVolume.SetUp succeeded for volume "serving-cert" Normal SuccessfulMountVolume 1m kubelet, localhost MountVolume.SetUp succeeded for volume "webconsole-token-zxjsp" Warning FailedCreatePodSandBox 13s (x2 over 36s) kubelet, localhost Failed create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "webconsole-7dfbffd44d-bz44s": Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:258: applying cgroup configuration for process caused \"No such device or address\"" Normal SandboxChanged 10s (x2 over 36s) kubelet, localhost Pod sandbox changed, it will be killed and re-created. https://gitlab.com/tom81094/bugs/raw/master/f28/docker-2:1.13.1-56.git6c336e4/openshift-web-console-logs https://gitlab.com/tom81094/bugs/raw/master/f28/docker-2:1.13.1-56.git6c336e4/oc-cluster-logs ditto here; can be reproduced by docker run --rm --cpu-shares=128 fedora:28 bash Please see: https://github.com/projectatomic/runc/pull/10 Which fixes this problem. (NOTE: While the backport of the single commit/PR seems to be enough, it's probably best to look at backporting more, since there were other changes around that code. Perhaps a whole refresh of upstream "runc" would be good there.) Cheers, Filipe I'm seeing this when installing/running openshift origin 3.9.0 on Fedora Atomic Host release candidate. This is blocking future releases of FAH. I tracked down the problem to this change: ``` # rpm-ostree db diff a5f1234a302fb064f67f09afe8ddd9cbac524a406a257a562fd18000dac99ba8 cefc79e6ea4d7e5eec51a32c00e1ecd6ca678d322406fecd347bc9c49e5d5255 ostree diff commit old: a5f1234a302fb064f67f09afe8ddd9cbac524a406a257a562fd18000dac99ba8 ostree diff commit new: cefc79e6ea4d7e5eec51a32c00e1ecd6ca678d322406fecd347bc9c49e5d5255 Upgraded: docker 2:1.13.1-51.git4032bd5.fc28 -> 2:1.13.1-56.git6c336e4.fc28 docker-common 2:1.13.1-51.git4032bd5.fc28 -> 2:1.13.1-56.git6c336e4.fc28 docker-rhel-push-plugin 2:1.13.1-51.git4032bd5.fc28 -> 2:1.13.1-56.git6c336e4.fc28 quota 1:4.04-5.fc28 -> 1:4.04-6.fc28 quota-nls 1:4.04-5.fc28 -> 1:4.04-6.fc28 selinux-policy 3.14.1-29.fc28 -> 3.14.1-30.fc28 selinux-policy-targeted 3.14.1-29.fc28 -> 3.14.1-30.fc28 Removed: oci-register-machine-0-6.1.git66fa845.fc28.x86_64 systemd-container-238-8.git0e0aa59.fc28.x86_64 ``` An example of a container not getting started is one of the glusterfs daemonset containers. Here is a snippet from oc describe: ``` Warning FailedCreatePodSandBox 7m (x16287 over 5h) kubelet, 10.0.12.155 Failed create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "glusterfs-storage-mlpdl": Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:258: applying cgroup configuration for process caused \"No such device or address\"" Normal SandboxChanged 2m (x16532 over 5h) kubelet, 10.0.12.155 Pod sandbox changed, it will be killed and re-created. ``` Any chance this is SELinux related? Mrunal, this is another bz about the cgroup fix that went into runc :/ https://github.com/projectatomic/runc/commit/99a2d0844a013541744154a07380422a073c4926 docker-1.13.1-59.gitaf6b32b.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-c2e93d5623 docker-1.13.1-59.gitaf6b32b.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-993659ebfd Ran an openshift cluster on top of docker-1.13.1-59.gitaf6b32b.fc28 using ostree ref `fedora/28/x86_64/atomic-host ` in repo `https://dustymabe.fedorapeople.org/repo/` fixes it for me docker-1.13.1-59.gitaf6b32b.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-c2e93d5623 docker-1.13.1-59.gitaf6b32b.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report. docker-1.13.1-59.gitaf6b32b.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-993659ebfd docker-1.13.1-59.gitaf6b32b.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report. |