1457912 – /var/lib/docker mount does not persist across reboot.

Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1457912 - /var/lib/docker mount does not persist across reboot.

Summary: /var/lib/docker mount does not persist across reboot.

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Fedora
Classification:	Fedora
Component:	docker-storage-setup
Sub Component:
Version:	26
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	---
Assignee:	Vivek Goyal
QA Contact:	Fedora Extras Quality Assurance
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2017-06-01 14:03 UTC by Dusty Mabe
Modified:	2017-06-24 03:06 UTC (History)
CC List:	3 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2017-06-24 03:06:01 UTC
Type:	Bug
Embargoed:
Dependent Products:

Attachments	(Terms of Use)
verbose output (8.07 KB, text/plain) 2017-06-02 16:07 UTC, Dusty Mabe	no flags	Details
View All

Description Dusty Mabe 2017-06-01 14:03:01 UTC

Description of problem:

/var/lib/docker mount does not persist across reboot. 

On first boot:

```
[root@vanilla-f26atomic ~]# cat /etc/sysconfig/docker-storage-setup 
# Edit this file to override any configuration options specified in
# /usr/share/container-storage-setup/container-storage-setup.
#
# For more details refer to "man container-storage-setup"
STORAGE_DRIVER=overlay2
CONTAINER_ROOT_LV_NAME=docker-root-lv
CONTAINER_ROOT_LV_MOUNT_PATH=/var/lib/docker
GROWPART=true
[root@vanilla-f26atomic ~]# lsblk
NAME                          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda                           252:0    0   41G  0 disk 
├─vda1                        252:1    0  300M  0 part /boot
└─vda2                        252:2    0 40.7G  0 part 
  ├─atomicos-root             253:0    0    3G  0 lvm  /sysroot
  └─atomicos-docker--root--lv 253:1    0 15.1G  0 lvm  /var/lib/docker
vdb                           252:16   0   20G  0 disk 
[root@vanilla-f26atomic ~]# 
[root@vanilla-f26atomic ~]# 
[root@vanilla-f26atomic ~]# journalctl -u docker-storage-setup
-- Logs begin at Thu 2017-06-01 13:43:34 UTC, end at Thu 2017-06-01 13:43:52 UTC. --
Jun 01 13:43:36 localhost.localdomain systemd[1]: Starting Docker Storage Setup...
Jun 01 13:43:37 localhost.localdomain container-storage-setup[747]: CHANGED: partition=2 start=616448 old: size=83269632 end=83886080 new: size=85366784,end=85983232
Jun 01 13:43:37 localhost.localdomain container-storage-setup[747]:   Physical volume "/dev/vda2" changed
Jun 01 13:43:37 localhost.localdomain container-storage-setup[747]:   1 physical volume(s) resized / 0 physical volume(s) not resized
Jun 01 13:43:37 localhost.localdomain container-storage-setup[747]:   Logical volume "docker-root-lv" created.
Jun 01 13:43:37 localhost.localdomain systemd[1]: Started Docker Storage Setup.
[root@vanilla-f26atomic ~]# cat /etc/fstab 

#
# /etc/fstab
# Created by anaconda on Wed May 31 16:12:35 2017
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/atomicos-root /                       xfs     defaults        0 0
UUID=e980060d-953c-41f7-a660-123676f6cfcd /boot                   ext4    defaults        1 2
```


Version-Release number of selected component (if applicable):
[root@vanilla-f26atomic ~]# rpm -q docker
docker-1.13.1-7.git14cc629.fc26.x86_64
[root@vanilla-f26atomic ~]# rpm-ostree status
State: idle
Deployments:
● fedora-atomic:fedora/26/x86_64/atomic-host
             Version: 26.51 (2017-05-31 15:32:01)
              Commit: 770ae054136d69c69e81f8860be99290472371e20bed2abdce1c1a9468259935

This also happens on rawhide with docker-1.13.1-13.git51eb16e.fc27.x86_64.



How reproducible:
Always.


Steps to Reproduce:
1. Boot an image from here: https://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170531.n.0/compose/CloudImages/x86_64/images/
2. Let the image boot.
3. Perform reboot. 
4. Observe /var/lib/docker is not mounted.

Comment 1 Vivek Goyal 2017-06-01 15:39:14 UTC

Can you provide container-storage-setup logs after reboot.

Comment 2 Vivek Goyal 2017-06-01 17:29:27 UTC

After reboot are you running docker or not? This will be mounted when docker runs which in turn will run docker-storage-setup and then it will mount.

I just tested on fedora 26 and it works for me.

Comment 3 Dusty Mabe 2017-06-01 17:47:10 UTC

@vgoyal - looks like I needed one more step to my reproducer:

Steps to Reproduce:
1. Boot an image from here: https://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170531.n.0/compose/CloudImages/x86_64/images/
2. Let the image boot.

2a. run a container ` docker run -it registry.fedoraproject.org:fedora:26 /bin/bash` 

3. Perform reboot. 
4. Observe /var/lib/docker is not mounted.


Here are my logs for docker-storage-setup for 3 boots. 1st boot was initial boot. 2nd boot came up fine. On 2nd boot i created a container. On 3rd boot /var/lib/docker/ is not mounted. 

```
[root@vanilla-f26atomic ~]# journalctl -u docker-storage-setup
-- Logs begin at Thu 2017-06-01 17:41:34 UTC, end at Thu 2017-06-01 17:44:20 UTC. --
Jun 01 17:41:36 localhost.localdomain systemd[1]: Starting Docker Storage Setup...
Jun 01 17:41:37 localhost.localdomain container-storage-setup[757]: CHANGED: partition=2 start=616448 old: size=83269632 end=83886080 new: size=85366784,end=85983232
Jun 01 17:41:37 localhost.localdomain container-storage-setup[757]:   Physical volume "/dev/vda2" changed
Jun 01 17:41:37 localhost.localdomain container-storage-setup[757]:   1 physical volume(s) resized / 0 physical volume(s) not resized
Jun 01 17:41:37 localhost.localdomain container-storage-setup[757]:   Logical volume "docker-root-lv" created.
Jun 01 17:41:37 localhost.localdomain systemd[1]: Started Docker Storage Setup.
-- Reboot --
Jun 01 17:42:15 vanilla-f26atomic systemd[1]: Starting Docker Storage Setup...
Jun 01 17:42:15 vanilla-f26atomic container-storage-setup[727]: NOCHANGE: partition 2 is size 85366784. it cannot be grown
Jun 01 17:42:15 vanilla-f26atomic container-storage-setup[727]:   Physical volume "/dev/vda2" changed
Jun 01 17:42:15 vanilla-f26atomic container-storage-setup[727]:   1 physical volume(s) resized / 0 physical volume(s) not resized
Jun 01 17:42:15 vanilla-f26atomic systemd[1]: Started Docker Storage Setup.
-- Reboot --
Jun 01 17:43:48 vanilla-f26atomic systemd[1]: Starting Docker Storage Setup...
Jun 01 17:43:49 vanilla-f26atomic container-storage-setup[721]: NOCHANGE: partition 2 is size 85366784. it cannot be grown
Jun 01 17:43:49 vanilla-f26atomic container-storage-setup[721]:   Physical volume "/dev/vda2" changed
Jun 01 17:43:49 vanilla-f26atomic container-storage-setup[721]:   1 physical volume(s) resized / 0 physical volume(s) not resized
Jun 01 17:43:49 vanilla-f26atomic container-storage-setup[721]: mount: special device /dev/atomicos/docker-root-lv does not exist
Jun 01 17:43:49 vanilla-f26atomic container-storage-setup[721]: ERROR: Failed to mount volume docker-root-lv on /var/lib/docker
Jun 01 17:43:49 vanilla-f26atomic systemd[1]: docker-storage-setup.service: Main process exited, code=exited, status=1/FAILURE
Jun 01 17:43:49 vanilla-f26atomic systemd[1]: Failed to start Docker Storage Setup.
Jun 01 17:43:49 vanilla-f26atomic systemd[1]: docker-storage-setup.service: Unit entered failed state.
Jun 01 17:43:49 vanilla-f26atomic systemd[1]: docker-storage-setup.service: Failed with result 'exit-code'.
```

Comment 4 Dusty Mabe 2017-06-02 16:07:24 UTC

Created attachment 1284484 [details]
verbose output

verbose `set -x` output was requested for when this happens on startup. Here is the output.

Comment 5 Vivek Goyal 2017-06-02 19:57:49 UTC

PR to fix this merged upstream.

https://github.com/projectatomic/container-storage-setup/pull/245

Comment 6 Dusty Mabe 2017-06-05 19:40:26 UTC

thanks @vgoyal - let me know when this makes it into an RPM that we can test.

Comment 7 Fedora Update System 2017-06-07 13:29:48 UTC

docker-1.13.1-16.git51eb16e.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-4b5b8b0479

Comment 8 Fedora Update System 2017-06-07 15:10:43 UTC

container-storage-setup-0.5.0-1.git9b77bcb.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-87e474598d

Comment 9 Fedora Update System 2017-06-07 23:11:23 UTC

docker-1.13.1-16.git51eb16e.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-4b5b8b0479

Comment 10 Fedora Update System 2017-06-07 23:11:26 UTC

container-storage-setup-0.5.0-1.git9b77bcb.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-87e474598d

Comment 11 Fedora Update System 2017-06-13 14:56:45 UTC

docker-1.13.1-17.git51eb16e.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-84372bb930

Comment 12 Fedora Update System 2017-06-15 03:01:42 UTC

container-storage-setup-0.5.0-1.git9b77bcb.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

Comment 13 Fedora Update System 2017-06-15 13:56:54 UTC

docker-1.13.1-17.git51eb16e.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-84372bb930

Comment 14 Fedora Update System 2017-06-15 14:20:54 UTC

docker-1.13.1-18.git27e468e.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-740382b040

Comment 15 Fedora Update System 2017-06-16 22:24:28 UTC

docker-1.13.1-18.git27e468e.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-740382b040

Comment 16 Fedora Update System 2017-06-19 19:53:53 UTC

docker-1.13.1-19.git27e468e.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-36d05500ee

Comment 17 Fedora Update System 2017-06-21 06:22:01 UTC

docker-1.13.1-19.git27e468e.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-36d05500ee

Comment 18 Fedora Update System 2017-06-24 03:06:01 UTC

docker-1.13.1-19.git27e468e.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.