Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1412728

Summary: Bind mount on /var/log seems to be over shadowed
Product: Red Hat Enterprise Linux 7 Reporter: Mohamed Ashiq <mliyazud>
Component: oci-systemd-hookAssignee: Mrunal Patel <mpatel>
Status: CLOSED ERRATA QA Contact: Martin Jenner <mjenner>
Severity: urgent Docs Contact:
Priority: high    
Version: 7.3CC: amurdaca, cww, ddarrah, dwalsh, gouyang, hchiramm, lsm5, mliyazud, mpatel, pprakash, rcyriac
Target Milestone: rcKeywords: Extras
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: oci-systemd-hook-1:0.1.4-9.git671c428.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-01-17 20:46:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1412281    

Description Mohamed Ashiq 2017-01-12 16:10:50 UTC
Description of problem:
We have a systemd container on which we have to have a bind mount on the '/var/log/<something>'. This worked fine before this release. This bind mount is strict requirement for us because of which we are hitting issue in our setups. 

Version-Release number of selected component (if applicable):
# rpm -qa | grep docker
cockpit-docker-126-1.el7.x86_64
docker-common-1.12.5-14.el7.x86_64
docker-1.12.5-14.el7.x86_64
docker-client-1.12.5-14.el7.x86_64
docker-rhel-push-plugin-1.12.5-14.el7.x86_64

# rpm -qa | grep systemd
systemd-219-30.el7_3.7.x86_64
systemd-sysv-219-30.el7_3.7.x86_64
oci-systemd-hook-0.1.4-8.git45455fe.el7.x86_64
systemd-libs-219-30.el7_3.7.x86_64

# rpm -qa | grep systemd
systemd-libs-219-30.el7_3.6.x86_64
systemd-219-30.el7_3.6.x86_64

How reproducible:
Always

Steps to Reproduce:
# docker run -d -v /var/log/something:/var/log/something:z rhel /usr/sbin/init
b8a202e69e364f0cee13a7127bc97a7ad9f55e5827e5462b5bc4ed09b1f12f74
# docker exec -it b8a202e69e364f0cee13a7127bc97a7ad9f55e5827e5462b5bc4ed09b1f12f74 bash
/]# df -h
Filesystem                                                                                         Size  Used Avail Use% Mounted on
/dev/mapper/docker-8:17-67158286-1c93270178ee4a8845a0b69c0729a516ffd9dc712d0bf7c2cb160021f02c94b0   10G  247M  9.8G   3% /
tmpfs                                                                                               24G     0   24G   0% /dev
tmpfs                                                                                               24G     0   24G   0% /sys/fs/cgroup
/dev/sdb1                                                                                           40G  1.2G   39G   3% /etc/hosts
shm                                                                                                 64M     0   64M   0% /dev/shm
tmpfs                                                                                               64M  236K   64M   1% /run
tmpfs                                                                                              4.0E     0  4.0E   0% /tmp
tmpfs                                                                                              4.0E  8.0K  4.0E   1% /var/log
/]# ls /var/log/
btmp     journal/ wtmp

/]# mount       
/dev/mapper/docker-8:17-67158286-1c93270178ee4a8845a0b69c0729a516ffd9dc712d0bf7c2cb160021f02c94b0 on / type xfs (rw,relatime,seclabel,nouuid,attr2,inode64,sunit=1024,swidth=1024,noquota)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,seclabel,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/net_prio,net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio,net_cls)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/cpuacct,cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime,seclabel)
/dev/sdb1 on /etc/resolv.conf type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/sdb1 on /etc/hostname type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/sdb1 on /etc/hosts type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,seclabel,size=65536k)
/dev/sdb1 on /run/secrets type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/sdb1 on /var/log/something type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,seclabel,size=65536k,mode=755)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=4503599627370496k)
tmpfs on /var/log type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=4503599627370496k)
/dev/sdb1 on /var/log/journal/552de008be2c0a1364cbacfc32ef526f type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/pids type cgroup (ro,relatime,pids)
cgroup on /sys/fs/cgroup/blkio type cgroup (ro,relatime,blkio)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (ro,relatime,net_prio,net_cls)
cgroup on /sys/fs/cgroup/cpuset type cgroup (ro,relatime,cpuset)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (ro,relatime,hugetlb)
cgroup on /sys/fs/cgroup/memory type cgroup (ro,relatime,memory)
cgroup on /sys/fs/cgroup/perf_event type cgroup (ro,relatime,perf_event)
cgroup on /sys/fs/cgroup/devices type cgroup (ro,relatime,devices)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (ro,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/freezer type cgroup (ro,relatime,freezer)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=28,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
configfs on /sys/kernel/config type configfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)


Please let me know if you need more information.

Comment 1 Daniel Walsh 2017-01-12 16:12:55 UTC
What version of oci-systemd-hook do you have installed

rpm -q oci-systemd-hook

Comment 2 Mohamed Ashiq 2017-01-12 16:16:25 UTC
# rpm -qa | grep oci
oci-register-machine-0-1.11.gitdd0daef.el7.x86_64
oci-systemd-hook-0.1.4-8.git45455fe.el7.x86_64


In the description First systemd version is on the host and second one is on container.

Comment 3 Daniel Walsh 2017-01-12 16:36:16 UTC
Mrunal, we are overmounting the directories on top of existing volume mounts and on /run/secrets.

First question?  Do you remember why we are mounting a tmpfs on /var/log?

When we mount over /run we should probably tar up the contents from the directory and put them into the tmpfs of at least mv any mount points off of /run into the newly mounted tmpfs.

# docker run -ti -v /var/log/dan:/var/log/dan:z fedora mount | grep /var/log
/dev/sda2 on /var/log/dan type ext4 (rw,relatime,seclabel,data=ordered)
tmpfs on /var/log type tmpfs (rw,nosuid,nodev,relatime,context="system_u:object_r:container_file_t:s0:c118,c249")
/dev/sda2 on /var/log/journal/184b22d455aafb6c9d56ce1f79cf3d20 type ext4 (rw,relatime,seclabel,data=ordered)

I can work on a fix, but I want  your opinion.

Comment 4 Mrunal Patel 2017-01-12 17:53:21 UTC
Dan, the /var/log tmpfs was for journald logs. I think we can do the same dance of MS_MOVE that we do for /run mounts. We move mounts temporarily using MS_MOVE and then put them back in place for anything over /run. We can do the same for mounts specified by the user over /var/log.

Comment 6 Guohua Ouyang 2017-01-13 02:54:10 UTC
tested oci-systemd-hook-0.1.4-9.git671c428.el7.x86_64.rpm, the fix works.

1. Reproduced the bug 
# mkdir /var/log/test
# docker run -d -v /var/log/test:/var/log/test:z rhel /usr/sbin/init
44f4d3a237c94b6f0b870f972f9adf638407827103283a19fccb55433495fd7b
# docker exec -it 44f4d3a237 bash
# ls /var/log
btmp  journal  wtmp
# docker stop 44f4
44f4
# docker rm 44f4
44f4

2. 
# rpm -Uvh oci-systemd-hook-0.1.4-9.git671c428.el7.x86_64.rpm 
Preparing...                          ################################# [100%]
Updating / installing...
   1:oci-systemd-hook-1:0.1.4-9.git671################################# [ 50%]
Cleaning up / removing...
   2:oci-systemd-hook-1:0.1.4-8.git454################################# [100%]

3. 
# docker run -d -v /var/log/test:/var/log/test:z rhel /usr/sbin/init
2c46308f2981e72fd19378da393e8719faaee85cdeec597377bb73b13c5b8133
# docker exec -it 2c46308 bash
# ls /var/log
btmp  journal  test  wtmp

The /var/log/test dir is there.

Comment 8 Humble Chirammal 2017-01-13 06:46:09 UTC
Thanks Dan for your quick help on this!! We are also validating the fix from our side and will update this bug accordingly.

Brew link : https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=12348885

Comment 9 Daniel Walsh 2017-01-13 12:31:55 UTC
Awesome.

Comment 10 Humble Chirammal 2017-01-13 14:59:02 UTC
Good news!! Gluster Container deployment is working as expected and we confirm that, the reported issue at our end is fixed with above mentioned build. Once again thanks a lot for quick help on this, much appreciated.

Comment 13 errata-xmlrpc 2017-01-17 20:46:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0117.html

Comment 14 Red Hat Bugzilla 2023-09-14 03:37:20 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days