Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1557356 - etcd.service fails to start: missing system user and rpm scripts
Summary: etcd.service fails to start: missing system user and rpm scripts
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: etcd
Version: 28
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Jan Chaloupka
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-03-16 13:26 UTC by Jan Pazdziora
Modified: 2018-06-16 20:15 UTC (History)
7 users (show)

Fixed In Version: etcd-3.2.16-2.fc27 etcd-3.2.16-2.fc28
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-06-16 19:31:48 UTC
Type: Bug
Embargoed:

Attachments (Terms of Use)

Description Jan Pazdziora 2018-03-16 13:26:31 UTC 
Description of problem:

Attempt to start etcd.service fails.

Version-Release number of selected component (if applicable):

etcd-3.2.16-1.fc28.x86_64

How reproducible:

Deterministic.

Steps to Reproduce:
1. dnf install -y etcd
2. systemctl start etcd
3. systemctl status etcd

Actual results:

Job for etcd.service failed because the control process exited with error code.
See "systemctl status etcd.service" and "journalctl -xe" for details.

● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2018-03-15 14:23:09 EDT; 7s ago
  Process: 10730 ExecStart=/bin/bash -c GOMAXPROCS=$(nproc) /usr/bin/etcd --name="${ETCD_NAME}" --data-dir="${ETCD_DATA_DIR}" --listen-client-urls="${ETCD_LISTEN_CLIENT_URLS}" (code=exited, status=217/USER)
 Main PID: 10730 (code=exited, status=217/USER)

Mar 15 14:23:09 machine.example.com systemd[1]: etcd.service: Service hold-off time over, scheduling restart.
Mar 15 14:23:09 machine.example.com systemd[1]: etcd.service: Scheduled restart job, restart counter is at 5.
Mar 15 14:23:09 machine.example.com systemd[1]: Stopped Etcd Server.
Mar 15 14:23:09 machine.example.com systemd[1]: etcd.service: Start request repeated too quickly.
Mar 15 14:23:09 machine.example.com systemd[1]: etcd.service: Failed with result 'exit-code'.
Mar 15 14:23:09 machine.example.com systemd[1]: Failed to start Etcd Server.

Expected results:

No error, service running.

Additional info:

The same steps pass on Fedora 27.
Comment 2 Jan Chaloupka 2018-03-16 21:43:02 UTC 
I am not able to reproduce it with etcd-3.2.16-1.fc28.x86_64 nor with etcd-3.2.16-1.fc29.x86_64 on my f24 local machine. Can you provide output of journalctl -xeu etcd.service ?
Comment 6 Martin Pitt 2018-03-20 08:05:44 UTC 
This is trivial to reproduce on a freshly installed Fedora 28, I see it in our new Cockpit test images, too. The error is simply this:

# journalctl -ocat -b -u etcd
Starting Etcd Server...
etcd.service: Failed to determine user credentials: No such process
etcd.service: Failed at step USER spawning /bin/bash: No such process
etcd.service: Main process exited, code=exited, status=217/USER

# getent passwd etcd
#

On Fedora-27, the "etcd" system user gets created by the %post script:

# rpm -q --scripts etcd
preinstall scriptlet (using /bin/sh):
getent group etcd >/dev/null || groupadd -r etcd
getent passwd etcd >/dev/null || useradd -r -g etcd -d /var/lib/etcd \
	-s /sbin/nologin -c "etcd user" etcd

Whereas in Fedora 28 it seems to have lost all of its scripts:

# rpm -q --scripts etcd
Comment 7 Martin Pitt 2018-03-20 08:23:23 UTC 
I played around with this a bit. In particular whether the %pre script could be avoided with `DynamicUser=yes`, but that also seems to fail:

    audit[2647]: AVC avc:  denied  { write } for  pid=2647 comm="(bash)" name=".pwd.lock" dev="dm-0" ino=4226691 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:passwd_file_t:s0 tclass=file permissive=0
    etcd.service: Failed to update dynamic user credentials: Permission denied

So this is a bug in Fedora 28 that breaks DynamicUser= in general. However, after `setenforce 0` starting etcd just hangs completely, and so does `getent passwd etcd`. So perhaps let's not get too fancy just yet :) (I'll file a separate bug about this).

I ran the groupadd/useradd commands manually, and now it gets a little further, but still fails:

    cannot access data directory: mkdir /var/lib/etcd/default.etcd: permission denied

which isn't too surprising as the directory is owned by root:

# ls -ld /var/lib/etcd/
drwxr-xr-x. 2 root root 6  9. Mär 15:56 /var/lib/etcd/

In Fedora 27 there is no chown/chmod in the %pre or %post scripts, so I wonder how that worked there. What works is to not ship /var/lib/etcd/ in the rpm, and let systemd create it with correct ownership at startup:

    [Service]
    StateDirectory=etcd
Comment 8 Fedora Update System 2018-03-20 11:06:47 UTC 
etcd-3.2.16-2.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-d9254d3db6
Comment 9 Fedora Update System 2018-03-20 11:07:11 UTC 
etcd-3.2.16-2.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-713528f10c
Comment 10 Fedora Update System 2018-03-20 14:55:08 UTC 
etcd-3.2.16-2.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-713528f10c
Comment 11 Fedora Update System 2018-03-20 19:39:16 UTC 
etcd-3.2.16-2.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-d9254d3db6
Comment 14 Martin Pitt 2018-04-03 11:37:16 UTC 
Confirming that on current F28 etcd.service starts correctly. Thanks!
Comment 15 Fedora Update System 2018-06-16 19:31:48 UTC 
etcd-3.2.16-2.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.
Comment 16 Fedora Update System 2018-06-16 20:15:09 UTC 
etcd-3.2.16-2.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.
Note You need to log in before you can comment on or make changes to this bug.